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ABSTRACT 

Thi s report documents the proceedings of a conference 
on the validation of exemplary educational programs, products, and 
practices presented by the Regional Exchange of the Southwest 
Educational Development Laboratory (SEDL/RX) ; It is different from 
other conference syntheses, as it includes the process for creating 
the conference as well as the conference content and the evaluation 
of it. The objectives of the conference were to give participants the 
opportunity to explore the state of the art in validation; to provide 
the time and the environment for each SEDL/RX state team to work on 
its bwn_ validation concerns ; arid_tp_ arrange a fbruiti arid begin to pool 
the collective knowledge of the SEDL/RX region regarding validation. 
Papers were presented by_William Hinze, Christine Dwyer, Bud_ 
Grossner , John Osborne , Preston C. Kronkosky , Jack Lumbley, Bill 
Reaves, and Sharon Adams. The conference evaluation consisted largely 
of determining if the objectives were attained through questionnaires 
and monitoring state validation efforts. (DWH) 
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FOREWORD 



The SEDL Regional Exchange (SEDL/RX) Project has provided information and 
technical assistance services since October 1976 to educators in five states: 
Arkansas, Louisiana, New Mexico, Oklahoma, Texas, and, since April 1979, Mississippi. 
It is one of seven Regional Exchanges in the nation-wide Research & Development 
Exchange (RDx) , funded by the National Institute of Education , which lists as a 
major goal the dissemination of information about educational research and development 
(R&D). To assist in accomplishing this goal , the SEDL/RX staff designed and sponsored 
the Regional Validation Conference in Austin, Texas on April 19 and 20, 1979, In an 
effort to record and pass on to others some of the experience and knowledge that 
was shared during that day and a half meeting, this document, A Regional Conference 
on Validation: How, What, and Why was compiled. 

This conference , like others sponsored by the SEDL/RX, provided opportunities 
for sharing, communication, and growth among researchers, clients, and users of the 
knowledge and products of educational research and development. 

dames H. Perry 
Executive Director 
August 1979 
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PREFACE 



This document may be different from other conference syntheses; as it includes 
the process for creating the conference as well as the conference content and the 
evaluation of it. The SEDL/RX staff recognizes that some readers may be more 
interested in process than product * while others would rather concentrate oil 
content. For that reason, we direct you to the following sections so that you may 
follow your interests: 



Planning information 



Planning Summary 



Conference Preparation Documents 



Section I 



Conference Content Documents 



Section II 



Conference Evaluation Documents 



Section III 
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PLANNING SUMMARY 



6ri April 19 and 20, 1979, the Regional Exchange of the Southwest Educational 
Development Laboratory (SEDL/RX) presented a conference on the validation of 
exemplary educational programs* products * and practices. This conference, the 
"Regional Val idation Conference," as it was called , brought together State Educa- 
tion Agency (SEA) personnel f rom six states , the US0E Region VI, and resource 
people from arcund the nation to study various issues in val idation and to work 
in detail on the topics which most concerned the individual states. 

This paper is more than a report of the proceedings. It represents an 
attempt to document not only what happened , but why. The SEDL/RX staff hopes that 
by detai 1 ing the process that preceded the conference as well as the conference 
itself and its evaluation results, we can provide a guide for others who may attempt 
the same or a simi lar project. We hope to save a few steps , avoid the reinvention of 
any more wheels , and perhaps answer two major questions , "When the attempt is made 
to disseminate information through a conference, can any effects be identified?" 
and , "Can these effects be attributed to the conference?" 

There are certain problems inherent in such an attempt. The SEDL/RX staff 
wondered if our evaluation forms were valid enough to get arc accurate reading of 
what happened. That led us to ask whether any paper form can truly evaluate an 
event or process, which led to the ultimate, if existential , question, can anything 
ever be eval uated? 

This is now known as the SEDL/RX Chinese Box Approach to Evaluation, supported 
by such axioms as: 

• everything is relative, 

• all solids are really liquids, i.e. don't take anything 
for granite* 

• it's always something else, 

• things are never as they seem. 
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Despite this brief descent to the Bottom Line of "Why bother?" the SEDL/RX 
staff has rallied, and is determined to attempt this documentation. 

The Raison d'@tr -4> 

The seed for sponsoring a validation conference was sown by the SEDL/RX 
Advisory Board members during the staff's work on creating a Regional Program File. 
In theory, the file would contain information on exemplary educational programs in 
each of the states of the SEDL/RX service area: Arkansas , Louisiana, New Mexico, 
Oklahoma, Texas, and later, Mississippi. In practice, the staff and Advisory Board 
members realized that recommending programs for the file would be difficult until 
state validation procedures were developed or refined. It became apparent, then, 
that an opportunity to examine the elements of effective state validation procedures 
would not only be valuable for the states , but also would be a necessary first step 
toward creating a Regional Program File, At the November 15, 1978 meeting of the 
SEDL/RX Advisory Board, a regional conference on validation was, therefore, suggested. 
Needs Sensing 

Janis O'Driscoll Schechter , Dissemination Specialist, became the SEDL/RX staff 
coordinator for the validation conference. The first step was to survey each state 
to determine as precisely as possible their specific needs. To do this, the staff 
wrote to six people on January 16, 1979: the USOE representative and one representa- 
tive from each of the five SEA 1 s.* The SEA representatives who received the needs 
sensing questionnaire were members of a task force established previously to work 
with Sharon Adams, SEDL/RX Dissemination Specialist, to create the Regional Program 
File. Each task force member was directly involved in some way with the use of 
exemplary programs in his/her state (see a copy of the request letter and questionnaire 
and a list of people to whom these were sent on pages 15-18); The letter 

*the Mississippi SEA did not join the SEDL/RX service area until April, 1979 and so 
was not included in planning for the conference. The first opportunity for the 
Mississippi SEA to participate in any SEDL/RX activities was at the April 18, 1979 
Advisory Board meeting, brie day before the validation conferences) 
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described the background for the proposed conference and the rationale: 

It appears that SEA's are becoming more and more interested in 
file building which includes awareness and recording of the 
state's, exemplary programs—many of which are rion-JDRP and non- 
Title IV-C programs and projects • As thoughts turn to evidence 
of effectiveness and replicability* questions arise: What valida- 
tion procedures and strategies are appropriate? What have others 
done in validation? 

The letter suggested a two-day conference at which a team of 5-8 people per 
state, designated by each Advisory Board member, would come to Austin to share vali- 
dation concerns and strategies both as states and as a region. 

The questionnaire enclosed with the letter asked for information" on the current 
status of the SEA 1 s validation procedures; whether the SEA was satisfied with its 
methods; what might be changed; issues a state team might want to consider at a 
conference, suggestions for resource people and materials; and provided an open 
question for other ideas respondents might have. 

Respondents were requested to return the questionnaire to the SEDL/RX in 10 
days, by January 26. They were also notified that they would receive a follow-up 
call on January 22, to request suggestions for specific dates and to discuss any 
problems encountered with the questionnaires. 

The follow-up calls not only served as a reminder to send in the questionnaire, 
but also allowed the respondents to say what they might possibly have been uncom- 
fortable putting in writing. It became apparent during the phone calls that, 
although possible dates for the conference suggested in the letter were mid-March 
or early April , a more convenient time would be mid-April , specifically April 19 
and 20. It also became apparent that state team groups of 3-5 were more appropriate 
than the suggested 5-8. 
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Identified Needs 

When the questionnaires were returned, SEDL/RX staff synthesized the informa- 
tion, looking for elements the states had in common and for possible regional trends. 
One conference could not effectively answer a large number of disparate concerns, so 
common ground was sought. 

The questionnaires/telephone calls revealed that Arkansas had a tightly 
structured procedure based on the IVD process; Louisiana had no validation procedure 
but had a commitment from the Dissemination Task Force of its Special Purpose Grant 
to create one; New Mexico had no state validation procedure but did have a second 
draft for a proposed process which the team would bring to get assistance in 
reviewing it and recommendations for models and implementation procedures; Oklahoma 
had a statewide validation process, used IVD forms, and was interested in refining 
details such as site visits and reporting systems; Texas had a statewide validation 
procedure for individual ized instruction programs and was contemplating its expansion 
to content areas rather than teaching techniques; 
Objectives, Outcomes, A udience, Bate 




Once these needs were established, specifics such as audience, objectives, out- 
comes , and resource people all began to fall into place. 

The objectives and expected outcomes were written for an audience of SEA per- 
sonnel responsible for creating or implementing the state's validation process. 
This audience was subject to the approval of the SEDL/RX Advisory Board and approval 
was received through telephone calls. The Advisory Board eventually chose the 
specific people who would attend the conference and notified the SEDL/RX of their 
choices. 

By February 1 , the dates of April 19 & 20 were established for the conference and 
objectives and outcomes were written; While the conference was designed as a 
regional forum, the objectives did not include attention to regional validation 
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procedures: Rather; a regional approach to the conference seemed the most logical 
method of bringing together the regional units to learn about strengthening them- 
selves individually and getting some perspective of themselves regionally. As 
the evaluation shows* however, interest in a regional validation procedure did 
emerge. 

Human Resources 

Consultants on the needs sensing questionnaire received favorable responses 
in general, and a few others were suggested. The SEDL/RX staff wanted the confer- 
ence's resource people to provide both national and state perspectives and so, 
using the questionnaire as a guide, proceeded to call possible consultants and 
confirm their availability. During February and March, the staff confirmed the 
services of Dr. William Hinze of the U.S. Office of Education's Division of Educa- 
tion Replication, to discuss his involvement in the creation and revision of the 
IVD handbook, Sharing Educational Success . (Hinze brought multiple copies of the 
document in final draft.) Ms. Christine Dwyer, of the RMC Research Corporation, 
agreed to speak about the IVD process from the perspective of her study of five 
states' validation procedures. Mr. Bud Grossner , Manager of LEA services for the 
Illinois State Department of Education, would discuss Illinois' validation procedure, 
and Dr. John Osborne, Supervisor of Experimental and Demonstration Programs for the 
Michigan State Department of Education, agreed to describe Michigan's validation 
process (Grossner and Osborne were secured through the support of Sharon Koenigs of 
the CEHREL/RX) . In addition, Dr. Preston C. Kronkosky, director of SEDL's Field 
Services and Dissemination Division, which houses the RX, and Mr. Jack Lumbley, 
evaluator for the division, were asked to provide their perspectives about living 
through the JDRP process. Dr. Ben Harris, professor of educational administration 
at The University of Texas at Austin, was contracted to serve as a participant/ 
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observer. Harris' purpose was to listen to each speaker; be ah unobtrusive 
observer in each state team meeting, report to the large group the second 
morning on similarities, differences, underlying themes, and implications for 
the SEDL/RX in addressing state concerns. Because some of the SEDL/RX staff had 
speculated whether a regional validation process would be an issue, Harris was 
also asked to provide an analysis of potential planning steps toward regional 
validation. Contract letters to consultants included lengthy details of dates* 
fees, rationale, state perspectives revealed in the questionnaire and purposes 
and lengths of requested presentations. (See sample consultant contract letter, 
page 27. j 
Material Resources 

In addition to human resources, printed resources were also located during 
February and March. Linda Reed, director of the Research and Development Interpre- 
tation Service at CEMREL, St. Louis, had agreed in January to write a synthesis 
document on the state-of-the-art of validation as reflected in current literature. 
SEDL/RX staff sent her a brief bibliography of relevant resources, and continued 
contact with her during the creation of the paper. SEDL/RX staff also requested, 
in February , a copy of THE STUDY OF THE IDENTIFICATION, VALIDATION , DISSEMINATION 
(IVD) PROCESS (the results of Christine Dwyers ' research) from the U.S. Office of 
Education, but a copy did not arrive until after the conference. Dwyer brought 
multiple copies with her to the conference. Plans were also made to provide a 
display table of materials available through the SEDL/RX Resource Center (see pages 
99-107 for information on the display and papers made available by presenters) . Plans 
also called for a preconference packet mailed to participants (detailing travel informa- 
tion, what to expect and not to expect, and their responsibilities), an information 
packet at the conference (see page 97), and a resource area for materials difficult 
to duplicate. All teams were expected to bring a work plan, that is, some idea of 
what they wanted to achieve while there and how to do it. 
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Collaboration 

Co 1 laboratibri with CEMREL's Sharon Kdehigs , Director* Training Program for 
Women and Minorities and CEMREL/RX staff member, also began in February and 
continued through the conference. The CEMREL/RX had been interested in the 
validation issue and was considering a conference of its own. When the CEMREL/RX 
staff learned about the SEDL/RX's plans for a conference, the CEMREL/RX provided 
financial support to bring consultants John Osborne and Bud Grossner to Austin, 
arid to allow Sharon Koenigs to attend. These three representatives of CEMREL's 
Regional Exchange hoped to gather ideas and perhaps a conference model for 
CEMREL/RX' s clients. 
Agenda 

By March 3, a tentative agenda was ready: Each morning would begin with a 
half hour of coffee, Juice, pastries , and conversation. At 9 a.m. on Apri 1 19, 
states would introduce themselves with a brief description of their work on valida- 
tion. They would be followed by national perspectives, a multi-state perspective, 
two state models from outside the region, and then a report of first-hand experience 
with a JDRP hearing. 

After lunch, states were allotted the afternoon to work on their own projects 
with the support of one consultant serving as a resource to the group. An optional 
presentation on evaluation procedures and techniques was scheduled for 3 p.m. 

Beginning at 8 a.m. April 20, the tentative agenda planned for a panel discussion 
by all the consultants on questions provided to them first by Dr. Ben Harris and then 
by participants. Information on the National Education Practices File would be 
provided by Sharon Adams ,* then the remainder of the morning would be devoted again 
to state team work and to reports of their work. The conference would end at noon. 



*The SEDL/RX was chosen as one of 12 pilot sites for this file, a creation of 
Bibliographic Retrieval Services of New York designed to fill the need for a 
nationwide system for collecting , organizing, and communicating information about 
educational programs , practices , and materials . 
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The final agenda did indeed follow the tentative agenda, with the addition of 
question/answer sessions following each presentation (see page 37). Mr. Bill Reaves, 
former school evaluator and current intern with the Texas Association of School Admin- 
istrators, agreed to present the optional session on evaluation from the LEA perspective 
(see pages 39-85 for texts of presentations). As it turned out, state groups were so 
involved in their work that no one attended the optional session. 
Evaluation Plans 

At a February 23 meeting, the SEDL/P.X staff discussed progress on conference 
details, including an evaluation plan. Discussion centered on previous SEDL/RX- 
sponsored activities which the staff knew had affected participants, but for which 
there was no documentation. The staff wondered if what appeared to be cause-effect 
relationships could ever be accurately traced, but decided to try. The evaluation 
plan which resulted called for the administration of an evaluation form at the end 
of the conference, a second one six weeks later, and the use of an objective parti- 
cipant/observer to observe each state team and provide a perspective for the region. 

In c idition, SEDL/RX staff members would each be assigned to work With One 
state team to get a better understanding of its progress and observe their activities 
with the resource person assigned to the group. The staff would also listen for 
informal comments about the conference made in group work or in conversation. 

This evaluation plan expanded on April 4, when the SEDL/RX staff met to discuss 
evaluation procedures. If we were to know if the conference had any immediate effect, 
we would first need to know how participants felt before the conference began: A 
pre-test was therefore suggested and the SEDL/RX Research Assistant in evaluation 
designed one consisting of 12 questions (see page 113). The questions asked for 
participants' opinions about state validation procedures and various elements of 
them, such as replicabil ity, on-site visits, and the training of validation teams. 
The pre-test also asked about the establishment of a regional validation procedure; 
The same 12 questions were then added to the original conference evaluation form, 
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creating a post-test to determine if the conference had any immediate effect dh 
opinions. The pre-test was administered the morning of the first day. It was 
intended to be given immediately upon convening, but was unfortunately overlooked 
until about one hour into the conference. The post-test (see page 115) contained 
34 questions, including four open-ended questions with space for written answers. 
In addition* presenters were asked to complete a 13-question evaluation form (see 
page 117). Results of the formal evaluation begin on page 123. 

Plans for a follow-up evaluation six weeks after the conference were delayed by 
the intervening requirement to write a proposal for renewed funding of the SEDL/RX. 
In late July, a form was mailed to all participants requesting information about 
their use of data or materials from the conference (see page 121). In the meantime, 
however, informal followup via telephone calls and requests for services indicated 
that the conference had made an impact. In New Mexico, for example, a third draft 
of the state's validation procedures was sent to Jack Lumbley for his review. 
Lumbley met with the New Mexico team in August to create the final version. In 
Texas, the state team wrote a paper at the conference outlining the pros and cons 
of extending its validation process to content areas. At the May 21 meeting of the 
Demonstration Schools Network Advisory Committee, the paper was discussed and the 
decision was made to expand the process. 

On May 11, Dr. Ben Harris, the participant/observer, met with SEDL/RX staff to 
share his perceptions of the conference (see page 145). 
The Site 

The site for the conference was relatively neutral— the 5th floor conference 
room at SEDL. Having the conference at SEDL provided a unifying elements-partici- 
pants could become more acquainted with the SEDL/RX home, be away from the distractions 
of an SEA, and also be close to their hotels and other downtown Austin attractions for 
their free time. 
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Communication 

Throughout the planning period^^cTose contact was continued with the task 
force * the SEDL/RX Advisory Board, the participants, the resource people* arid the 
SEDL/RX staff. As each stage progressed, the conference coordinator notified the 
affected person(s) by telephone, letter, or conversation. 

Oh April 5, for example, a letter (see page 33) was mailed to participants 
giving them details about the hotel , conference site, reimbursement procedures, 
and travel costs. The letter explained that each team heeded to choose a spokes- 
person to present the state 1 s introduction the first morning, and asked them to 
bring materials describing their state 1 s validation work to be displayed on the 
materials table, shared with other states, or given to them. Original plans to 
distribute Linda Reed's paper on the state-of-the-art with this "letter were changed 
when the paper's revisions were not ready. The letter therefore informed partici- 
pants that the paper would be available at the hotel when they checked in. Parti- 
cipants were encouraged to read the paper, use it during the conference and give 
comments on it to SEDL/RX staff, either orally or in writing. 
Suwort Services 

One crucial , but often overlooked, element in conference planning is the role 
of support services for the conference coordinator. The SEDL/RX Senior Secretary 
was responsible for typing everything related to the event: contract and other 
letters, packet contents, agenda, evaluation forms , and so on. She was also 
responsible for name tags, arranging for refreshments , making and changing hotel 
reservations, occasionally providing transportation to and from the airport, and 
processing expense forms. She also typed this document. Without accurate, creative, 
and responsible ; upport services available, a conference is in severe trouble, if 
not doomed; 
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C o n~c I - u s l o n s~ 

The regional validation conference was designed to provide human and material 
resources, state team time and cross-team sharing; and ah opportunity to create or 
at least work toward a useful product. 

The conference also provided time for participants to hear resource people, 
to question them in a large group setting, to work with them on specific issues, and 
to take advantage of their expertise during the state team work. This provided the 
opportunity for airing various viewpoints. Keeping the state teams down to a maximum 
of five people provided manageable numbers both for the teams and for the conference 
as a whole. Scheduling the conference for a day and a half allowed participants to 
be away from their or f ices for as short a time as possible and still benefit from 
concentrated study of an important issue. The days could have been scheduled to 
begin earlier and end later, but relaxation time is also integral to work, and free 
time allowed participants, planners, and resource people to get to know one another 
more informally. 

In summary, we have found several core elements which are important in planning 
and conducti ng a conference: 

• conduct the needs sensing in a variety of ways to be certain final 
results will be valid. For example, if participants are different 
from the people who originally recommended the conference, check 
with participants for their viewpoint. In addition look for verifi- 
cation from, rel evant print sources if available; 

• know the levels of awareness of participants and match human and 
material resources to those levels; 

• base the conference on specific participant needs and organize it 
to meet those needs as efficiently as possible; 

• give yourself extensive planning time (4-6 months is not excessive) ; 

• establish frequent, informative, and accurate communication with 
participants and presenters; 
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get adequate funding to pay for sucb things as participant/ 
presenter travel costs, consultant fees* reproduction of materials, 
stipends, f bod \ and site costs. 

consider physical comfort as well .as intellectual stimulation. Provide 
adequate time for breaks, lunch, starting, and stopping; then stiek to 
the schedule; 

this conference was designed to include both regional and state heeds b 
so small- and large-group time was allotted; 

it is possible to give participants "homework" in advance, request that 
they finish it before they come, and expect them to do it. It requires 
persistent emphasis that the work is important and will be dealt with 
specifically, and it is then imperative that the assignment be used at 
the conference and not simply referred to or overlooked entirely; 

participants can determine their own activity for a period of time at 
the conference if the conference coordinator involves theiii in early 
planning and keeps in nearly constant touch; 

an objective participant/observer who is more observer than participant 
(in this case, Dr. Harris) can be valuable if care is taken to choose 
the person carefully; 

it is tempting to look forward to the close of a conference as the end 
of productivity on the topic. It is therefore easy to "lose" any 
hoped-for effects which might occur as a result by not following up 
with participants several weeks later. Evaluations administered the 
last day of a conference can only register immediate reactions. Follow- 
up evaluations provide participants time to consider new information 
with the distance of both time and space and also to implement new 
approaches. 
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I. PREPARATION FOR THE CONFERENCE 
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Southwest Educational Development Laboratory 
211 East 7th Street, Austin, Texas 78701 

512/476-6861 

danuary 16, 1979 



THIS LETTER SENT TO THE FOLLOWING PEOPLE: 
Dr.- Sara Murphy, Arkansas 
Br. Ron bearden, Louisiana 
Mrs. Susan Brown, New Mexico 
Mrs. Wilda Copeland, Oklahoma 
Mr; Vaughn Al dredge, Texas 
Mr. George Blassingame, USOE Region VI 



The SEDL/Regional Exchange has appreciated your help on the Educational 
Program File Task Force, and we would like to ask your advice again. 

The SEDL/RX Advisory Board has expressed interest in a two-day regional 
conference on validation procedures for exemplary programs ._ We are proposing 
to bring a team of people (5-8 persons per team, designated by the state's 
SEDL/RX Advisory Board member) into Austin to consider validation concerns and 
strategies. It seems appropriate to share and to learn both as a region, and 
as an individual state, so we've suggested a team approach; 

Why have a validation conference? It appears that SEA's are becoming more 
and more interested in file building which includes awareness and recording of 
the state's exemplary programs— many of which are non-JDRP and non-Title IV C 
programs and projects. As thoughts turn to evidences of effectiveness and 
replicability, questions arise: What validation needs do we have? What vali- 
dation procedures and strategies are appropriate? What have others done in 
validation? 

Answers to these questions are not easily found. The Exchange suggests that 
one way of finding these answers might be to consult as a region with selected 
resource people and sample state models, and then to take some time to work with- 
in each state group to concentrate on its own needs. But to find the right 
resource people, to review the most appropriate models, and to raise the most 
significant validation issues, we need your help. 

May I ask you to take some time to respond to the attached questions? We 
ask them because we are interested in designing a conference that you will find 
useful and worth your time to attend. You may prefer to attach already-prepared 
material in answer to some of the questions; at the same time, you may wish to 
write short answers to others. Please return your responses to me by January 26, 
if possible. 



y ■ 



! » ; i i. 



ERIC 



15 



23 



I am planning to call you on January 22 to sound you out on two things: 

• Dates suggested for the 2-day conference; week of March 19-23 
or the week of April 2-6; 

• Any difficulties or concerns regarding the attached questions, 

I will be glad to discuss any other matters of interest regarding this 
conference at that time. 

I appreciate your dealing with all of this paper, and I know your perspective 
will be truly helpful in designing this conference. 



Thank you for your time and consideration. 



Sincerely* 

Janis O'D. Schechter 
Dissemination/Linkage Training 



JS/ta 
Enclosure 

cc: Preston C. Kronkosky 
Sharon Adams 
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REGIONAL EXCHANGE 

PLANNING INFORMATION FOR SEDL/RX VALIDATION CONFERENCE 



1. Please describe the current status of validation procedures in your SEA. 
Is there a statev/ide process, or are there several processes in use? 



2. Is your current method of handling. validation satisfactory? What would 
you like to change, improve, or delete? 



3 - The SEDL/RX Advisory Board has expressed strong interest in a region-wide 
• gathering of people interested in validation. It has been proposed that a 
team from each state might consider validation issues^ in general, and its 
specific state needs in particular. Would a team from your state be 
interested in: (please check all that apply) 

□ A discussion of important validation issues and concerns, such as: 
(please list) 



□ Description of various validation models currently in use. 

□ Time for your state team to work on your own state validation concerns 
together, with a resource person if desired. 

□ Discussions concerning a regional validation procedure for the SEDL/RX 
Education Program File. 



The SEDL/RX has recently been selected as a pi 1 bt site for a test of the 
proposed National Program File._ Would a discussion of. our region's partici- 
pation in this pilot test be of value to you during this conference? 



Suggested resource people: Bill Hinze of OE, who is currently revising 
Sharing Educational Success, a description of the IVD process; Greg Benson, 
Director of the National Program File; John Osborne, Director of the Dissemi- 
nation and_ Utilization Division of the Michigan SEA; Joan Miller, Coordinator, 
Educational Programs and Studies Information Service: New York SEA; Richard 
Brickley for the Pennsylvania Diffusion Plan. Are there other resource people 
whom we should consider for specific tasks? 



Suggested resource materials: JDRP's kleabook^jbaring Educational Success. 
Descriptions of procedures in Alaska, Michigan, New York, Pennsylvania, and 
Texas. What other materials should be included? 



These are some of the possibilities for a regional conference on validation. 
What other items or concerns should we keep in mind as we go about planning 
the conference? 



OBJECTIVES FOR SEBb/RX VALIDATION CONFERENCE 



Td give participants the opportunity to explore the state of the art 
in validation by giving them access to: 

. knowledgeable and experienced people representing 
a variety of perspectives 

. material resources representing both policy and 
procedural aspects of validation issues 

To provide the time and the environment for each SEDL/RX state team 
to work on its own validation concerns: 

. with the opportunity for interaction with other 
experienced people 

. with a number of significant material resources 
close at hand 

To arrange a forum where participants from the SEDL/RX region, along 
with other resource people with varying viewpoints , may exchange 
experiences, information, and concerns about the establishment and/or 
maintenance of validation procedures. 

To begin to pool the collective knowledge of the SEDL/RX region 
regarding validation, and to begin to identify from that pool the 
essential elements of a validation procedure. 
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EXPECTED OUTCOMES OF THE SEBL/RX VALIDATION CONFERENCE 



All participants will have ...... 

. increased their familiarity with issues, options, 
concerns, and models in validation 



. increased their familiarity with validation 
procedures in the SEDL/RX region 



State teams will have ...... 

. engaged in some significant work on their own 
validation concerns 

. received some material resources which will be 
useful back home as well as at the conference 

. made contact with some .experienced people 
inside as well as outside the region who may 
be of further help in the future 

. formed a clearer notion of the ways the SEDL/RX 
could be of further assistance in their 
val idation work 



SEDL/Regiohal Exchange staff will have ...... 

. had the opportunity to assist clients in meeting 
validation needs 

. increased its understanding of specific validation 
issues in each of its client states 

; become better prepared to be of further assistance 



Resource people will have ...... 

. shared their perspectives on validation by being 
information-givers as well as listeners and 
helpers to individual states 

. gained an understanding of the resources and 
experience of SEA people in the SEDL/RX region 
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Southwest Educational Development Laboratory 
211 East 7th Street, Austin, Texas 787G1 

512/476:6861 



February 27, 1979 



gllltflM^TOiillliii i l^hih mMI' 

RMC Research Corporati on 
111 Bow Street 

Portsmouth, New Hampshire 03801 
Dear Christine: 

This is to follow up our phone conversation today regarding SEDL/Regional 
Exchange's Validation Conference in Austin, Texas on April 19-20. 

Our Exchange serves five states: Arkansas, Louisiana, New Mexico, Oklahoma, 
and Texas, and they are each in a different stage of development with regard to a 
statewide validation procedure.. Nevertheless , they have requested a regional 
validation conference which will meet both regional and individual state interests. 

We hope to make this conference a working conference. Each state has been 
asked to send a team of 3-5 people who can work on validation issues and problems 
confronting that particular state. Whi le the teams are in Austin, ue hope to 
present significant resources in validati on, and to provide the opportunity for 
each state team to work on its own with a resource person. 

I believe that the perspective that you have from the work you have done in 
validation would be of great interest to our states^ I also think that the 
individual state teams would profit from your assistance with their own state plans. 

If you are able to be a presenter/resource person at our conference, we would 
agree to pay for travel , meal , and lodgi ng expenses plus a consultant fee of 
/day. We would estimate those costs to be: 

Airfare from Manchester, N.R. to Austin, Texas $390.00 

Other travel costs (includes travel to and from 
airport). . .' 40.00 

Meals for 2 1/2 days .................. 45.00 

Lodging ........................ 70.00 

Consultant fee for 2 days onsi te and 1 day preparation . _ 

ESTIMATED TOTAL . . $ 

As required by our NIE grant, we would also need to have your social security 
number and a copy of your resume on fi le. 
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Ms. Dwyer 



February 27, 1979 



Please call me if you have any questions or eoneerns regarding any of the 
above. I look forward to hearing from you soon, and I hope that we will see you 
in Austin in April . 

Sincerely ,/~\ r\ * 



Janis (Vb. Schechter 
Dissemination Specialist 



JS/ta 

cc: Preston C. Kronkosky 
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Southwest Educational Development Laboratory 
211 East 7th Street, Austin, Texas 78701 

512/476-6S61 

March 28, 1979 



Dr. Ben M. Harris 

Department of Educational Administration 

College of Education 

Education Building 310 

The University of Texas at Austin 

Austin, Texas 78712 



Dear Ben: 

I am pleased to learn that you wi 1 1 be able to assist us with the SEDL 
Regional Exchange's (SEDL/RX) Regional Validation Conference/Workshop 
at SEDL, April 19-20, 1979. The enclosed yellow sheet identifies the 
Expected Outcomes of the SEDL/RX Validation Conference. The enclosed green 
sheet identifies the Objectives for SEDL/RX J/al idation_Cbnference. Also 
enclosed is a copy of the tentative agenda for the Conference. 

As we discussed, your role would be to: 



1 . attend the presentation [.sessions Thursday_morning and 

afternoon so that you would "hear" the same things that 

the SEA participants hear (our SEA's are Arkansas, Louisiana, 
New Mexico^ Oklahoma^ and Texas plus the appropriate 
Education personnel from OE Region VI); 

2. be an unobtrusive observer in each of the five concurrent 
individual state team time meetings (1:30-4:30 p.m.). and the 
Optional Session [if it materializes at 3:00 p.m.)> 

3. report to the group and SEDL/RX on Friday morning [during the 
8:30 a.m. - 9:30 a.m. session and/or the 11 :00 a.m. - 11 :40 a.m. 
session) on the questions or concerns that you observed/heard during 
the Thursday afternoon sessions ~ what were the concerns/questions 
common to all 5 SEA's; 4 SEA's, 3 SEA's; 

How were the SEA's similar? 
How were they different? 

What seemed to be common underlying themes? 

What are the implications for SEDL/RX in addressing the concerns 
on an individual state basis? on a region-wide basis? 

4. advise me (Friday afternoon or at a later time) as to what you 

see as our next steps in terms of Region-wide Validation Procedures. 
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Br. Beri_M. Harris 
March_28, 1979 
Page 2 



In addition, if your Thursday evening commitments permit, we would be 
pleased to have you as our gu^st. during the informal dinner arid the 
ensuing continuing discussions. 

For your involvement in the above, we are budgeting the foil owing amounts: 

$. 



2 days of consultative services at $ /day 
lunch $5.00 and dinner $9.00 on 4/19/79 

Total 



14.00 



$ 



Oh, by the way , our consultative services guidelines require. that we must 
maintain a written report for our files on the results of all consultations 
charged to a grant or contract. This report must include, at a minimum: 

1. The consultant's name, dates and hours of work, arid* amount 
charged to the grant; 

2. The consultant's social security number; 

3. A vita delineating the consultant's qualifications to 
perform the work/services requested; 

4. A signed Consul tative Services Contract (see the enclosed); 

5. The names of the grantee staff to whom the services are 
provided; arid 

6. The results and/or products of the consultation. 

Currently, we have sufficient information for items 1 arid 5 above. On or 
before April 19, 1979, would you please furnish us with the information 
required in items 2, 3, and 4 above. On April 20, 1979, would you. please 
furnish us with a copy of your analysis and/or synthesis (it doesn't have 
to be "smooth" or "polished"). That, plus our own documentation should 
satisfy the requirements of item 6 above. 

If at all possible, please contact me or Ms. Jan Schechter (512/476-6861 
ext. 270 or 304) during the period April 9-12, 1979, to further define your 
proposed role with us. We are looking forward to hearing from you and working 
with you. 

Sincerely, 




Preston C. Kronkosky 
Director 

Field Services arid Dissemination 



PCKrbd 
Enclosures 

cc: Ms. dan Schechter 
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March 22, 1979 



(This letter sent to Dr. William Hinze and Ms. Christine Dwyer) 



I am delighted that you will be able to be a resource person at our 
validation conference in April. It seems that interest in - our conference 
has been shown by at least three other SEA's outside our region, so v/e 
may have eight states and three Exchanges represented. • 

Our main focus is, of course, the teams from the five SEDL/RX states 
who are coming to the conference With all kinds of needs and concerns 
regarding validation. From the planning information v/e received, we 
determined that our people are interested in a state-of-the-art review of 
validation, especially as it pertains to the development of state 
validation procedures. Also, they are interested in having individual 
team time with a resource person to work on their own specific validation 
concerns. With these items in mind, we have. tried to design a one and 
one-half day conference that cur clients will find useful. 



Our major focus in the design of the conference is access to a variety 
of resources and the opportunity to work with those resources on an 
immediate problem or plan. I have enclosed our conference objectives, 
expected outcomes, and tentative agenda. Please call me with any questions, 
concerns; or suggestions you may have. 

In January, a planning questionnaire was distributed to selected 
e in our states. Allow me to review some of the things that were stated. 

Louisiana : No validation procedures is in effect now, but a commitment 
has been made by the Dissemination Task Force of Louisiana's 
Special Purpose grant to construct one. They are especially 
interested in "funding procedures, housing and staffing, 
and composition of the team." (I am assuming the last two 
items refer to site visits.) Louisiana is primarily 
concerned with an orientation to state validation procedures. 
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March 22, 1979 



New Mexico : "There is not a state 'validation process in New Mexico; 

however, the first draft describing such a process has. been 
written and reviewed by four staff members. As a result 
of that review, a second draft has been prepared. Please 
note, however, that this second draft has not been 
reviewed by SDE staff as yet.. Also the nomination and. 
self evaluation, forms have not been designed. I will be 
glad to bring these additional items with me to the 
conference arid share them with the other state represen- 
tatives, providing the information has been reviewed and 
approved by our SDE cabinet." "Our primary concern would 
be assistance in reviewing the draft of our proposed 
validation procedures and models and suggestions for 
implementing the procedures." (New Mexico's draft is 
enclosed) 

Oklahoma: Several people from Oklahoma responded to. our questionnaire, 
and I will attempt to summarize the remarks. Oklahoma 
does have a statewide validation process arid Uses the 
current IVD forms. Oklahoma representatives have expressed 
the following concerns: 

. "We would like to shorten the forms" 

. "We have a concern which relates to method for handling 
thgse_programs which appear on their face to be effective, 
but which have not been" handled so that statistical 
analysis can properly be applied" 

. Site visits: "length of visit, size of team" 



. "Establish better relationship between state validation 
results arid reaction of national group to state report" 

. "A need for a system for reporting so that reporting can- 
be compared without having identical validation procedures" 

Texas : Texas has a statewide validation procedure for validating 

programs using the individualized instruction teaching technique. 
They are now contemplating an expansion of that procedure 
beyond the one teaching technique arid are asking such questions 
as: 

. "What purposes are there for validation?" 

. "Should our expanded procedure build in the grooming of 
the program for national validation?" 
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March 22, 1979 



Texas : : "What kind of emphasis on replication makes sense when 

(coht. } there are no funds' for statewide model programs?" 

. "Should there be different kinds of evaluation" Is a 
site visit always necessary? 

. "What about limiting the scope of the statewide model 
program? 

. fill in the 'gaps' in the NDN collection? 
. limit program to one or two content areas? 
. or, don't limit the program at all?" 

. "What kind of follow-up should Be built in after 
validation?" 

As you have no doubt realized, I have not included information about 
Arkansas. Arkansas' team has a couple of options in planning its work focus 
for the conference, and it should have its plans completed by next week. I 
wil] send you their ideas and concerns as soon as these decisions have been 
made. I will also include material describing Texas' validation procedure 
in the next mai 1 ing. 

This is already the longest letter I have ever written, but let me add 

a few more items. We have reserved a room for you at the: 

Driskill Hotel 
117 East 7th Street 
Austin, .Texas 78761 
(512) 474-5911 

The Driskill does not provide transportation to and from the airport (only for 
Bran iff personnel), and the taxi one-way should be $4-$5. Let me know when 
you are arriving, and perhaps I can meet you at the airport. 

Thank you for your patience in reading this letter* and I am looking 
forward to working with you in April. 

Sincerely, 

Original sigrH l » 

Jam's O'D. Schechter 
Dissemination Specialist 
SEDL/Regidnal Exchange. Project 
(512) 476-6861, ext. 306 



JS/ta 

Enclosures 

cc: Preston C. Kronkosky 
Sharon Adams 
Ncncy Baker Jones 
Morrie Schulman 
Karen Olsen 



March 23, 1979 



(Sent to all conference participants) 



I aril pleased that you will be attending SEDL/Regi onal Exchange's 
Conference on Validation on April 19-20 in Austin, Texas. We. have tried 
to be responsive to the recommendations that our planning committee made, 
and hope you will find the conference profitable. 

I am enclosing a copy of the conference objectives, expected outcomes, 
and detailed agenda so that you can begin to prepare for the work in 
Austin. Our planning committee stressed that each state team be given 
time to work on its own validation concerns and needs and we have built . 
that into the conference agenda on Thursday afternoon.. All presenters in 
the morning session will be available to work with individual state teams 
in the afternoon. We will begin by tentatively assigning a resource person 
to a state, but arrangements can be made to give states access to every 
resource person it wishes to consult. Also, members of state teams may be 
of assistance to other states, if needed. 

Along with the guidance of experienced resource people, we will have 
several material resources for your use while in Austin. Everyone will 
receive a copy of a paper written for our conference by Linda Reed, Research 
and Development Interpretation Service, CEMREL, Inc. (St. Louis). .Linda's 
paper will present a state-of-the-art perspective on validation. In addition 
there will be a resource area containing validation models, reports, and 
sample forms which might be useful as each team works on its own plan 
Thursday afternoon. 

If the Exchange can be of any assistance to you as your team makes its 

workplan for the conference, please do not hesi tate to cal 1 me. Perhaps 
there will be certain documents that we can make available, or perhaps 
there are certain people that you know you will want tc have time with. 
In that case, let me know and that time will be arranged as conveniently 
as .possible. It might be a good idea for each state team to get together 
before corning to Austin to discuss the ways. you can use the time Thursday 
from 1:30-4: 30pm and Friday from 10:20-11 : 00am. You might want to be sure 
you will. have. all the documents you will need, and perhaps consult interested 
cithers who will not be able to attend the conference. 
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March 23* 1979 



We. see the conference as an opportunity, to provide access to some 

validation resources and to allow you time to see how useful those resources 
might be in your situation. Please let us know it there are other ways 
thlt we can support you. 

As conference time approaches, I will be mailing you some additional 
material plus information regarding SEDL's reimbursement procedures. If 
you have any questions, please call me. 

I look forward to working with you in April. 

Sincerely* 

Originil signed by 

Janis O'D. Schechter 
Dissemination Specialist 

JS/ta 

Enclosures 

cc: Preston 6. Kronkosky 
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April 5, 1979 



(Sent to all conference participants) 



There is actually a hint of warm weather in Austin, and hopefully it will 

continue through April 19-20. We are all looking forward to your participation, 

in the Validation Conference, and are trusting that the weather forecasters will 
give us some support. 

Enclosed is a copy of SEDL's reimbursement procedures. We have made 
reservations for you at: 

The Driskill Hotel 
117 East 7th Street 
Austin, Texas 78701 
Phone: (512) 474-5911 

The Driskill does not provide transportation to and from the airport (only for 
Braniff personnel ) , and the taxi going one-way should be $4 - $5. The Driskill 
is a 5 minute walk from SEDL. which is located at Southwest Towers, 211 E. 7th 
Street^ on the same side of the street as the hotel. 

The conference will be held in the 5th floor conference room. We will post 

signs near the elevators to direct you, and there will be several SEDL/Regional 
Exchange staff members on hand to greet you. 

I have a request of you. Would you collectively select someone from your 
state team to introduce the team members and give a brief, 5 -minute explanation 
of what your state is doing/or preparing to do in validation. Perhaps, also, 
you might indicate how you expect to u~,e your state team time. This will be our 
first activity Thursday morning (after the pastries of course! ) . 

A table in the conference room will hold resource materials available for 
your use during the conference. We hope that you might bring copies of materials 
that describe validation work in your state. Perhaps you have a description of 
your staters procedure, a catalog, or a policy statement explaining your 
perspective .Teams from other states may re from what you have 

already l^earned^ The materials may be loans for the duration of the conference 
or they may be hand-outs. Be sure to let me know how you would like your 
materials treated: I am enclosing a sheet describing a sampling of other 
materials we will have available. 

38 

33 



One more item. I have. made reference to a paper which Linda Reed, Research 
and Development Interpretation Services, CEMREL, has written for the conference. 
The paper, a state-of-the-art in validation, has just been received in first draft 
form. Revisions heed to be made in the paper, and I will not be able to mail the 
paper to you as I had hoped. I will, however, have your copy available to you when 
you check into the hotel Wednesday night. Please ask for it when you register. 
I think that you will find the paper to be a quick, readable stage-setting for 
conference, and I hope you have a chance to read it and use it during the conference. 

In any event, I look forward to seeing you soon, and working with you. 



Si nee rely, 




JS/ta 



Enclosures 



cc: Preston C. Kronkosky 
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REGIONAL EXCHANGE 

SEPL/RX VALIDATION CONFERENCE AGENDA 

April 19-20, 1979 



Thursday, April 19, 1979 

8: 30-9 :00am Get .Acquainted 

(coffee^ tea* juice* and pastries for atmosphere) 

9 : 00-9 : 20 Introducti ons 

Review of Agenda 

9: 20-9: 50am Dr. William Hinze, Division of Education Replication (0E) 

will speak about his revisions of the IVD Handbook, 
Sharing Educational Succ ess- 



9 : 50-10 :00am Quest ion /Answer period 

10:00-10:30am Ms. Christine Dwyer, RMC Research Corporation 

will speak on the IVD process from the perspective 
of her recent study of five states 1 validation 
procedures (for the National Association of State 
Advisory Counci Is) 

10:30-10:40am Question/Answer period 

10:40-10:50am BREAK 

10: 50-11 :05am Illinois' validation procedure 

Mr. Bud Grossner, Manager 
LEA Services 

Illinois State Department of Education 

11 : 05-11 : 10am Question/Answer period 

11 : 10-11 : 25am Michigan's validation procedure 

Mr. John Osborne, Supervisor 
Experimental & Demonstration Programs 
Michigan Department of Education 

1 1 : 25-11 : 30am Quest i on/Answer period 

11: 30-1 :00pm LUNCH 

1:00-1 :20pm "Living Through and Living With a Validation Procedure 1 ' 

Dr. Preston C. Kronkos ky, Di rector 

Field Services S Dissemination (FS&D) 

Southwest Educational Devel opment Laboratory (SEDL) 

Mr. JackLumbley, FS&D Evaluator 

Southwest Educational Development Laboratory 



1:20-1 :30pm Question/Answer period 



SEDL/RX VALIDATION CONFE-'NCE AGENDA 
(continued) 
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Thursday, April 19, 1979 
(continued) 

1 : 30-4: 30pm Individual State Team Time with Resource Person 



3:00pm OPTIONAL SESSION:. H A Vocal school district looks 

at evaluation" Mr. Bill Reaves, former eval uator 
with school district in Grand Rapids, Michigan and 
currently an intern with the Texas Association of 
School Administrators. 

Friday, April 20, 1979 

8:00-8:30am Eye-Opening Ceremony 

(coffee* tea* juice, and pastries will* be essential 
elements in the ritual) 

8: 30-9: 30am Panel of Conference Resource People—each panel 

member will be asked to respond to two-three 
pre-arranged questions, and then questions will 
open from the floor 

9:30-10:00am Sharon Adams will discuss SEBL/RX's participation 
as a pilot site in the National Practice File, as 
well as SEDL/RX 1 s Regional Program File pilot test 



10:00-10: 10arj Question/Answer period 
10:10-10:20am BREAK 

10:20-11:00am Individual State Team Time: 

Continue previous work A talk to resource people, 
discuss implications of the morning's sessions 



11:00-1 1:40am Conference recommendations: 

Based on experiences, resources, plans, what advice 
would you give to people working on validation 
procedures concerning issues, scope, essential 
elements , resources , purposes? 

11 : 40-12: 00 noon Closing Remarks (especially ones you won 1 t want 

to miss about reimbursement forms, conference 
evaluation, and rides to the airport) 
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PRESENTATION BY DR: WILLIAM HINZE 



Back in 1968-69 when we were deeply entrenched in ESEA Title III* 
Congress had given our program a commitment to do some dissemination and 
we wrote into all the programs that were funded federally a percentage 
or a certain sum of money — a line item — called dissemination. Some 
money was allocated and was supposed to be spent for that purpose. But 
we found that there was a lot of paper shuffling about other people 
visiting one another and we weren't so sure that some of the processes 
or products that were being distributed were really worthy of dissemination. 
We felt that there had to be some way that we could determine the value of 
the products which were being disseminated. 

So we began to write to the state departments and say, "What method, 
what way, do you determine whether or not a project has some worthwhile 
material to distribute to other colleagues across the state? 1 ' We found 
that there were maybe three or four states that tried to do something 
about determining the value of the products or the processes which were 
being funded federally but there was naturally no uniform way of determining 
that. 

So we began to poll our group and say, "Now look, whatever you're doing 
let's put it down on paper and give us a chance to look at it; why don't 
you check with some of your local project people to determine whether or 
hot they are interested in having some way of determining the value of 
their effort? 11 

I don't know to what extent you're familiar with the way Title HI 
seesawed back and forth from the federal administration to state administra- 
tion to partially state administered, partially federally administered, but 
finally it ended up with 15% of the funds retained at the US Office level 
for administration and 85% of the funds retained by the state. We thought 
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that it would be well to get some Uniform process arid so we had the LEA's 
and SEA's working with each other to develop what they thought was their 
way of evaluating or looking at programs-. 

I am not using the term "validation 11 yet because it wasn't really 
coined at this point. We really didn't know what we were looking for at 
that point either, and I can remember we inviied all the state education 
agencies' Title III people to St. Louis and we had our first opportunity 
to look at what was supposed to be a way of evaluating programs. To make 
a long story short it was a year and a half of real hard work cutting and 
pasting. We tore, literally cut and pasted, the book that wa's tentatively 
written in St; Louis and came up with a handbook. At this point the term 
"validation" came about and we got the Identification, Validation, and 
Dissemination Handbook—the IVB Handbook—and that in turn was sent but 
to the states. 

States reviewed it and sent us their comments for revisions and 
Ultimately it was printed in final form and disseminated to all the states. 
Then we had to develop a training program; We developed ten regions. 
They went back and trained their people within their region; We've trained 
366-466 people in the effectiveness/success area, which is basically the 
evaluation part of it, the resource specification, and exportabi lity . 

The handbook has probably gone through three revisions; this the 
fourth one. Initially it was geared for Title III, but then we decided 
that it was such a good document that we should include all other programs. 
So we took out all the references to Title III, other than a little history, 
and piloted it with five Title I programs in and around the Washington area. 
That was a real interesting experience because Title I at that point had no 

i 

objectives that were identified with any of their programs and the validation 1 

effort is based bri objectivity. So, Title I really had to write objectives j 
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for the project that they were piloting the process on. The process has 
been used for other programs. Career education has used it* but basically 
we're trying to get it universally acceptable for any program that has 
objectives and can get some hard data. 

We actually started with seven projects about 1968. We sent a 
consultant at World Wide Consultants in Salt Lake City about 150 projects' 
evaluation data. He reviewed it and then selected 25 of the programs, 
went to visit those programs, and ultimately came out with seven Title III 
programs out of the whole nation that he thought were worthy of validation. 
From there we got the idea to go on and expand the process to- what it is 
currently. Right now we must have about 200-275. We haven't had a great 
deal of activity in validation recently because the lack of emphasis on 
validation in our office has caused us hot be as active as we should have 
been. We haven 1 t had a chance to keep tabs on the states as to how many 
projects they have validated as such but it's closer to 275 than it is to 
200. 

Let me pass out these handbooks. Turn to the first page and in the 
middle of it in big bold print put DRAFT. This latest revision was 
generated in Seattle, Washington. This document as I said, was developed 
out in Seattle, and Dr. Al Ellwel 1 was part of bur team that worked on it 
as was Dr. Bill McDougal 1, a professor of education at the Washington State 
University in Pullman. Dr. Bill Davis, one of the state facilitators who 
is working in the Colorado facilitator function, and Leonard Nackman, the 
evaluator for the state of Minnesota, were also involved, We literally— 
again— cut arid paste, wrote on the blackboard and erased and put it baek 
up again arid came up with a document. I took it back to Washington, 
edited it and tried to put some finishing touches, then sent it out to 
what i call the Torrid 19. 
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A year ago we had a few bucks left to do some evaluation arid we wanted 
to get some first hand information from people who had been involved with 
validation over the years. So I contacted 25 people and 19 ultimately 
showed up. We spent two days looking over some issues and policy decisions 
that had to be made relative to validation and, based on that group's work, 
the revision reflects all of their attitudes. In May 1979, we're going to 
have that same group. Nackman, Davis, and McDougall are coming and we're 
going to put on some finishing touches. I'm going to share seme of those 
ideas that we are going to incorporate in this document and hopefully that 
will be it. Then we will go through the process of getting it cleared 
through our of f i ce of management and budget so that we can begin to do some 
planning for training people in this version. I am a little reluctant to 
say "this version" and yet we're trying to make this a little more compre- 
hensive and a little more rigid in its outlook. On page one, a statement 
will be identified indicating that anyone who participates in this activity 
or process is doing it on a strictly voluntary basis. If we had any 
inclination to make it mandatory, it would have to be checked much more 
rigorously by the of f i ce of management and budget. As such they have to 
have some indication that it is a voluntary process, that people who want 
to get involved in the validation effort are doing it of their own free will 
and that we are not imposing this great document on them. As you well 
know, those of you who have seen the previous document, this is not quite 
as formidable and I hope that will definitely be a plus. 

In the basic introduction, then, we have what the two validation 
programs are. The term "validation" is used interchangably with "JDRP" 
and "IVD" and I have a little problem with that myself. I don't know how 
we can resolve that; I'm hoping we can make some kind of delineation and 
not call them one and the same because they are not the same. JDRP (Joint 
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Dissemination Review Panel) came into existence about five or six years 
ago when Dr. John Evans approached OE Commissioner, Dr. Marling former 
superintendent of Pittsburg public schools with the idea of developing a 
kind of good housekeeping stamp of approval for the Office of Education to be 
put dh some practices or programs which the office felt were of value for 
dissemination. So it was first the Dissemination Review Panel, made up 
of Office of Education people. When the National Institute of Education 
came into existence and began looking at dissemination of activities, they 
felt it only proper to include people from that office as wel-1, so now 
there are 11 people designated from Hit and ll from 0E who make up a 
panel of 22 to pass judgment on programs; They only need seven for a quorum 
and that's generally what shows up. That's how the DRP became JDRP, the 
Joint Dissemination Review Panel. I'll get around to doing something about 
trying to identify the two different JDRP and IVD concepts; it's a 
conundrum and were going to have to work it out and I hope we can do it 
without jeopardizing one or the other activities. 

On page five we have a description of the state-level IVD process which 
involves those states which have determined that they want to have ownership 
of an IVD process. We had a request from the faci 1 i tator project out in 
Colorado for those states which had what they considered to Be their own 
IVD process. Colorado was going to make an analysis, so I identified 13 
that I felt had something concrete that they could send to the people in 
Colorado. Ultimately they did get nine and when it was cut and dried* the 
process wasn 1 t al 1 that different. It ' s basical ly the same IVD process 
with a few little frills and innovations and the major thing is that they 
were using in-state people as validators. We had previously considered 
that to be a no-no. We had had out-state people so that we wouldn't have 
to deal with nepotism, or influence, or whatever, but basical ly that was 
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the only difference* that more states had their own people doing the job. 
Michigan is probably the most organized. Basically it is the same except 
a little more strict with the people they deal with at the tEA level. That's 
basically what the state level IVD process is all about. 

Qri page six,, we're hoping that we can expand the kinds of activities 
that the JDRP will accommodate. They have accepted the idea that they can 
only deal with the traditional research design comparing two treatments. 
We're hoping that we can ultimately get them to accept process kinds of 
programs, programs that deal in the affective area that would- be great for 
the gifted, for instance. There are all kinds of programs that we feel 
should go through that panel so that we in turn could get them in our 
National Diffusion Network. 

On page nine, we had to acquiesce a great deal. We're going to permit 
states to use in-state people. The pressure has been applied from above our 
division that all states should be involved. If we have as many as 13 states 
which have their own programs with in-state people as validators, we need to 
incorporate them within our national effort. But what we're doing here in 
item #3 is to recommend highly that team leaders be selected from out of 
state, though in-state people can be used. 

The next page deals with the levels of validation. We're expecting 
that once a project is validated and it has the kind of material that is 
needed to go through the JDRP, that it would take that final step arid go 
through the panel for that ultimate good housekeeping stamp of approval . 
Then you as a state department could determine what level of activity you 
would accept: maybe a process or product kind of validation, which might be 
a criterion referenced test instrument, for instance, or a handbook, or a 
curriculum, or a teacher ' s guide. 
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On page 27 begins the team reporting section; we're going to ask that 
the team respond to the question, "Were the data of such a nature?" Notice 
we have "yes" and "no." We don't have a Likert scale in this. It's 
either/or and I think that we'll find it a little more effective this way 
rather than using the point-system. Did the data reflect statistical and 
educational significance? Was the impact on the educational area, of such 
a nature to warrant dissemination of a greater nature? You know, statisti- 
cally if yoU have a large universe it isn' t very difficult to get any statis- 
tical significance and it might raise the kids' reading level by a month 
for a two-year period of time which is net that great, but if you've got 
enough kids that you've tested, that might be significant. So, we're 
asking the people to look at this particular item more carefully. 

Oh page 34 we're going to address #3* that a plan for training staff 
exists. Now in most cases when you are looking at a project that has been 
in existence for maybe two or three years, their training manual is probably 
in the head of one of the project directors and we 1 re not going to make it 
quite as blatant that it exists, that is is planned. 

On page 40 is the final recommendation. We want the people to be as 
hard nosed as possible with this process; If a team says this project or 
practice is validated, we want them to be able to say, "You have the 
capability of passing the JDRP panel and it will go th rough." If they say, 
"No* we'll approve for state validation but it doesn't have the kind of data 
that we think the JDRP would accept," we would like to think that the team 
is going to have to get their halos shined up in good order and in some cases 
play God; But we want the group to be highly qualified. You'll see the 
document outlines some criteria for people who are to be trained in this 
process. 
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We expect to have as many as three to five people in each state who 
would be trained and under the circumstances you could ship them from one 
state to another. We would like to think that we could keep the group 
relatively small, highly qualified, and well trained. Let me stop and ask 
if there are any questions. 
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PRESENTATION BY MS. CHRISTINE DWYER 

Let me first mention the NASAC study and I will give you some summaries 
of the studies after I talk. 

As part of a private contracting consulting firm* I get involved in all 
manner of things and one of the topics that a number of us have been 
involved in rather heavily over the last year has been validation. We have 
been working with some of our New England states to develop specific valida- 
tion processes for some special topic areas. We worked in Massachusetts to 
develop a process which depends heavily on describing essential elements 
as well as evaluation effectiveness. 

Another Title I effort that we dealt with was in Vermont and that was 
a very different kind of process. There the identification of projects in 
validation was left to the contractor to avoid some previous problems that 
they had had. That again was within Title I and it was a product 

evaluation/validation rather than a program. 

And a third effort along those lines was to develop a process for 
validation of teacher corps projects within our teacher corps network on 
the east coast. 

Another area that 1 1 ve been particularly working on in the last 
year to year and a half has been tracing the experience of adopters over 
two or three years, the adopters of JDRP projects primarily, but also 
adopters of some in-state validated projects; We've been looking at the 
experience that the adopters have had in implementing a project, in 
replicating effectiveness, and also the retrospective opinion of the 
adopters about the selection process, about the degree to which the 
validated project met their expectations, the degree to which validation 
was in fact accurate was important to them, etc. So I have a lot of 
interest in that kind of experience, I'll share perhaps a little bit of 
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that with you as we go through the report. 

I am also, from a very selfish point of view* interested in validation 
because I act as ah external evaluator for several Title IV projects. I'm 
pulling for them and I'm expecting that their designs are strong enough to 
get them through our state processes as well as hopefully on to the JDRP. 
So from that angle I also have that selfish interest. 

The topic I am going to discuss now is the NASA6 study. There were 
three basic purposes of the study: one was to identify characteristics of 
projects which choose to enter the IVD and which successfully complete a 
validation process at a state level. With entering projects * we were 
looking particularly at evaluation experience, evaluation background and 
evaluation design. A second effort was to look at some aspects of the 
current IVD mechanism as it existed in the states, particularly with the 
focus on the validation of effectiveness. And a third purpose was describing 
some of the characteristics of in-state diffusion networks. I am going to 
concentrate on the first two purposes. 

Let me tell you what states we looked at. When we say "state" or "state 
department," we were talking to Title IV-people, to state facilitators, to 
in some cases, other people who had responsibility for dissemination from 
the state level. We visited Florida, Washington, Minnesota, New Jersey, 
and Colorado for that purpose. To get another perspective we visited some 
developer-demonstrator projects , which have been JDRP approved. All but 
one had also gone through its state validation process. The projects were 
Project Host, which is a reading project; the Seaton Hall preschool in 
Minnesota; New Model Needs from Ohio, which is a secondary values project; 
Project Stem from Colorado, a math project; and Project Active in New Jersey 
which is physical education for handicapped students. In addition to the 



52 



developers in each state, we talked to several adopters of that project* 
some in-s\ate adopters, and in many cases they were adopters who had 
adopted the project after its state-level validation, not as a result of 
its JDRP validation. So we tried to get the range from funding agent to 
developer to validation team to disseminator to adopter to get the whole 
spectrum. The generalizations that 1 1 11 share with you are not intended 
to be fully generalizable to all states or all experiences. They're a 
snap shot view of some locale. 

Let me list, with brief discussion, some of the characteristics of 
the projects choosing to enter IVD. One of the things we found in the 
states that we visited was that financial consideration played a major 
part. It was sort of a no- choice decision for a lot of projects; Projects 
had a sense that they wanted to maintain project activity either to do 
further development or to spread the good word, really believing in what 
they had done. In many cases, though money doesn't necessarily come directly 
from validation, validation certainly increases the potential that a project 
might be able to get some further dollars to disseminate, to do further 
development, or to spread the word, and that was true whether dissemination 
was really tied to the process or not. For example, sometimes at the local 
level a project was ■ motivated to go through validation to build its credibility 
within its own district, so that long term support in terms of dollars may 
have resulted from the LEA, not from an external source. Validation was 
the stamp of approval which might help that to occur. As competition has 
increased for devel opment funds for new projects or for spin offs of other 
projects, SEA's dollars for development of new projects have decreased. 

More money has gone into dissemination so that the competition, the 
dollars available for further development, is really keen and validation 
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becomes more important if it does increase the potential to garner funds 
from another source. IVD becomes, then, a route to funding support. 

Another reason, another characteristic of the projects that choose 
to enter the IVD is that they have project directors who are very special 
kinds of people. They are described to us by state level people as 
innovators, sometimes mavericks within the system, education entrepreneurs, 
people in education who had skills that in the past had not been required: 
public relations skills. They definitely stood out and the project 
director was certainly crucial in the decision to go ahead and enter the 
IVD process or to become validated; The actual decision may have rested 
with someone else within the system, but the project director's willingness 
to go ahead and ability to carry it through were really key factors. 
Incidentally, those qualities were ones that were frequently mentioned 
later on by adopters as qualities that attracted them to the project. I 
found that true in New Hampshire and Maine also in studying adopters. The 
public relations skills, the innovativeness, the charisma* all of those 
qualities attached to the project director have* for some adopters—I 1 11 
ad an editorial note— gotten in the way of looking at the projects, perhaps 
from an evaluator's point of view. 

Another characteristic was the role that state department consultants, 
particularly Title IV-C people, had in influencing the decision to enter 
validation. That was an area that we found varied tremendously among the 
states. In some states, the day-one decision to fund a project influenced 
whether that project would eventually go to validation or not. In other 
words, the decision to fund a project was almost the same as the decision 
that the project would go to IVD and all the necessary resources that would 
make that project ready with evaluation information with materials, whatever, 
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were provided along the way so that three years from the date of fundi rig » 
the project would be all set to go through. In some cases, the state level 
help came much later and almost at the appl i cation arid self-study phase. 
The decision to do that depended sometimes on the assumed expertise of 
the project director. Those without expertise might be given more help all 
along. 

One of the influences that that state level role had on the project, 
or I think the area where it was most obvi ous , was in the selection of 
objectives or the grooming or changing of objectives as the project developed, 
so that when the project was ready to go to IVD there were objectives and 
information to back up those objectives which would pass the process; thus 
the process influenced the kinds of objectives the project has. In some 
cases, people would tell us that as they got near the end they rewrote their 
objectives so that they would be more in tune with what they knew the 
process was demanding- A footnote here — the awareness of JDRP standards has 
tremendously influenced that role. The emphasis, whether it is rumor or 
fact, is on measured statistical significance. That rumor is spread out to 
the hinterlands and to some degree the implication is that initial funding 
decisions may be based on what is likely to be validated; 

I find in New Hampshire that when I go out for a first visit to a 
project, the first thing that a project will say is* "Can you help me get 
through JDRP?" And I'll say, "Well, how about getting your project first? 
Let's talk about what it is that you wane to do." You may have run across 
that experience in some cases too. There are a lot of feelings out there 
about what is required now. 

Another characteristic of projects choosing to enter was that the 
previous experience with evaluation was really minimal. The staff really 
had expertise, but this was not true in all the projects we visited, of 
course. Evaluation was usually .an externally contracted part of the project; 
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it tended to be viewed as a necessary requirement as a way of assuring 
validation. 

Another characteristic was that projects that decided to enter were 
sometimes motivated by the desire to impress the folks on the home front, 
to be recognized locally for what they had done. As the IVD processes 
existed there was a lot of room in PR at the local level arid many projects 
took advantage of this or saw this as a main advantage. 

The final characteristic that I'll mention revolves around dissemina- 
tion . Projects that choose to enter IVD were very naive about what 
dissemination really was all about. The view from project directors was 
that it was important to be ready to disseminate. Rarely was anyone ready 
to disseminate and state department people indicated to us that projects were 
rarely rejected for those reasons. So there is a feeling that you should 
be ready, you should have some manuals ready, you should have thought 
about your training, while in fact* projects rarely were ready and that 
didn't matter. Now, I have to say that we're talking about projects 
that went through IVD five years ago and the sophistication of what dissemi- 
nation is has changed so rapidly that I would hesitate to say that was 

true today. ; 

Let me mention some of the characteristics of the current IVD 
mechanism. First, the application phase* the self study phase. In retro- 
spect, many project directors felt that the most valuable part of the IVD 
process was doing the application, that marshalling all the forces and 
getting all the material and documentation together to complete the applica- 
tion was the thing that was most important in later going to JDRP and 
in packaging materials for dissemination. It was a chance to get everything 
together and once people had gotten everything together there was also an 
expressed feeling that we did something here that was important; it was an 
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attitude that carried them through. In some cases in the application phase, 
the self study phase* the state department role is very intense. A lot 
of consultant help was provided* particularly evaluation from the state 
department. The key activity at that application phase is selection of 
the objectives. I don't think anybody would quarrel with that* that's not 
really a surprise. One thing that we found was that people collected a 
great deal more evaluation information that they used in an application. 
It was selecting the particular kinds of measures, the particular kinds 
of results, that were felt to be most helpful in getting through validation. 

What about perceptions of evaluation needs for validation? The 
discussions that we had most frequently centered around the evidence of 
effectiveness part of the validation procedures. Most of the project 
directors we talked to have gotten a lot of experience. They're well 
known in' the National Diffusion Network, but they still have a mystique 
that they associate with evaluation arid most of them felt very strongly 
that evaluation had little to do with the relative merit of their project: 
"There were evaluation results and they were important * we'll let somebody 
else put it together for us, but I know the project is good for other 
reasons." I'm sure that's the kind of dual feeling you've run into from 
a lot of different quarters. So the project director looked at that 
evidence of success val idation in that way. 

State department people were rather confortable with evaluation as a 
tool. They saw the evidence of effectiveness as an important criterion, 
a sort of a base line way to enter the door and the other kinds of informa- 
tion, the testimonial , the observational > the attitudinal , as frosting on 
the cake, more than the project di rectors did. 

Sf-^e faci li tators , in the long run, felt that initial validation as 
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it related to evidence of effectiveness was hot that important. They made 
their decisions or made their comments to people about projects not based 
on evidence of effectiveness but on informal feedback from the network 
that exists among facilitators, the information that they'd gotten from 
adopters that had tried the project. Rarely did anyone look back to see 
what the effectiveness claim was. 

Adopters, on the other hand, found the claims of effectiveness of 
minimal interest. I've talked to lots of adopters for this study and I've 
done interviews with probably 50 adopters in New Hampshire and Maine over 
the last year* and very rarely is an adopter interested in the claims of 
effectiveness. Frequently you'll hear adopters say, "I knew the project 
was validated," and pursuing that I'd ask, "Well, what does validation mean 
to you?" I don't think I found more than one or two people who had any 
notion of what the validation process entailed. Someone had put the seal 
on it; somehow it got there and somebody said okay and it was someone at 
the state level . 

The sophistication about validation is one of those areas that may be 
changing. The first selection of a program for an adopter may not be a 
very good way to judge the level of adopter knowledge right now. What 
adopters were interested in when selecting a project were a couple of 
things. I don't think there were any surprises. The key thing is 
critical local need—that the project they were going to select matched the 
needs they felt existed in the home district of the LEA. And if you probed 
a little, frequently you found that there had been some crisis, however 
minor, that had precipitated the search for a new project. Be it low 
reading scores, changing staff, a newspaper expose of what was happening in 
the schools as far as drugs, vandalism, or whatever. .Some crisis precipitated 
a search for a project. That was the key element that adopters identified 
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as important to them; Costs were secondary to adopters. If you think of 
some of the eriteria of exportabi lity that relate to the validation process 
you might think of validation exportabi lity in terms of what adopters 
were saying they look for. Adopters are looking for a match to need. 

Another thing they 1 re looking for is the availability of some ongoing 
relationship with that developer* with that project that is going to be 
a demonstrator. That they're not going to be like a textbook company, 
that there's going to be somebody there that they can go to when* they have 
a problem. That was very frequently mentioned. 

A third thing that adopters indicated as important to them was the 
readiness of materials arid the appeal of the materials that were available 
from the developer. Uniqueness was not important to adopters. Adopters 
said things like, "It's something I could have done myself if I had the 
money or time." The first time I heard that, I was surpri sed, but I've 
since heard it from a lot of people and a lot of contacts. I labeled 
that as the one-step-away phenomenon . It would have to be pretty close to 
what we could have done at home but they went ahead an did it so let's 
take it arid do it* let's riot re-invent the wheel. UriiqUeriess was viewed 
by state faci litators and state department people as limiting, and in some 
cases a liability for diffusion. Of course it makes sense. A project that 
is going to be unique is going to have fewer potential adopters; 

A number of the people that we tal ked to also felt it was important 
before maki ng an adoption decision to be able to make an cnsi te visit. 
New remember we were talking to in-state adopters of in-state projects so 
in most cases that was feasible to do. We weren't talking to people from 
Texas who had adopted an Ohio project. But my other experience in New 
Hampshire and in Maine indicates to me that thi s is not true. So I dori ' t 
know what to do with that. We have many, many projects in New Hampshi re 



arid in Maine that adopted from other parts of 'the country and no one ever 
the ■■ . of making an onsite visit. But in the NASA6 study the adopters 
that we talked to felt it was important to see it at the home base, make 
sure it exists there; New Englanders just have more confidence in the 
country, I don't know. 

Let me mention a few characteristics of the evaluation experience of 
projects that had gone through state validation. They tended to have 
fairly traditional quasi-experimental design; some had control groups; the 
data that they used was objective data, reporting either norm referenced 
or criterion referenced test information. As I mentioned before, many of 
them had collected attitudinal, observational information. They didn't 
report it c the time of the validation. The kind of evaluation needed 
for the IVD they viewed as not the type of evaluation generally useful at 
the LEA level. This was a special kind of thing; This isn't how we 
usually evaluate the worth of our programs; This may be why we're having 
trouble getting adopters to evaluate their replications. 

I've mentioned exportabil ity from the A/f level already. Some of the 
considerations they thought were important were the potential match between 
a project and LEA. But it's really the market place that takes over and 
the things I mentioned before are the market place concerns: need, ongoing 
support, materials, cost. "Let us decide about the cost," was what AA's 
were saying. "I'm not going to reject on the basis of cost unless it's 
outlandish; We* 11 make those decisions;" [*Adopti on/Adaptation) 

The question of validation affording identifiable advantages to adopters 
was one thing we wanted to find out—basically people did feel that it did 
provide consumer protection. I characterize it along a couple of lines; 
Orie thing it does, and I think this is very often skipped over, is outline 
and define the project; The nature of the validation process— the application, 
the self study phase, the explanation to other people— helps to define the 
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elements that are critical to the project. It seems so simple, but those 
of you who have visited a number of projects, or talked to a number of 
people who are trying to express to you what their projects are all about, 
will have some sympathy with what I'm saying. What are those key ingre- 
dients? Validation helps the project answer that. It ensures the develop- 
ment of reliable documentation, so therefore it reduces risk. It doesn't 
eliminate risk, but it reduces risk. There are significant cost reductions 
in terms of development, and validation identifies some particular persons 
who themselves become sharable products. One thing that validation has 
done has been to identify some key people. 

Another issue that we dealt with in the executive summary has been 
incorporated into the new IVD manual and that is the relationship between 
IVD and JDRP. That was the constant topic of conversation and one that 
lots of people were upset about: too much duplication of effort. Other 
people were saying the IVD is a much stronger process than the 6BRP beeause 
it allows for the onsite visit and it's real people talking to real people. 
Alternatives were discussed and across states that was a definite area of 
interest. 

One other which I'll mention briefly is the life span of validated 
projects. In the states that we visited 5 validation had been going on for 
awhile and projects that were validated quite a few years ago were still in 
the market place. And there was concern expressed by some state level 
people, some projects , and some state facil itators that a life span be 
established for projects* that re-validation occur after a particular time 
if a project wanted to continue disseminating. I think it was John Row, 
the project director of New Model , who said, "If you were a textbook, after five 
years you'd consider a textbook revision . Give us some time to revise arid 
come back again. " On the other hand some people from states arid state 



facilitators were saying that lots of hewer and great things have been 
developed in the meantime; five years ago was five years ago; let the bid 
guys but and allow some room for hew people to come in. So those were 
the life span issues and the JDRP and IVD were things that we didn't intend 
to talk about or ask questions about but were on lots of people's minds. 
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Let me lead irito my remarks by telling you a story about the Cracker 
Jack Dog Food Company* the biggest dog food company in the United States. 
They called their annual meeting and had- an auditorium full of people. 
The president is up before the group, giving them a very motivated speech, 
and he asks the group, "Who has the best dog food in the United States?" 
Arid the group stands up and hollers, "Cracker Jack Dog Food!" Then he 
says, "Who has the best sales force in the United States in the dog food 
business?" Arid they stand up again and yell* "We do, Cracker Dog Food!" 
Then he asks another question "Who has the best management?" Arid the 
Cracker Jack Dog Food people stand up and holler, "We do!" And the presi- 
dent says, "Then would somebody tell me why our sales are so lousy?" A 
meek little person stands up in the audience and says, "Because the damn 
dogs don't like it." 

I use that because I may be different from many of the other people 
that you're going to talk to today. In my role with the state agency, I 
see myself as the advocate of the dog, or the advocate of LEA, the user. 
In Illinois we have a newly created department called LEA Services which is 
a staff of about 100 people with the sole responsibility of providing 
technical assistance to school districts. We're very committed to that 
effort and we find that we are quite often thrown into the role of an 
advocate for the LEA's. Many times that advocacy calls for us to take on 
our own agency. In fact quite often, to be quite frank with you; 

But it's Because of that effort to deliver services to the LEA's that 
we are extremely interested in dissemination. And of course you can't spend 
much time in dissemination and not get involved in the validation process. 
We don't have any problems with the IVD process; we use it in Illinois. I 
think we have 38 or 39 projects in the Illinois Diffusion Network. We 
utilize those processes and value them, but frankly in working with the LEA's 
we find that holding to just those two processes limits us too much. 
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What I have to say to you deals more with promising practices. The 
state of the art in Illinois right now is dealing with that question. And 
we're just starting to deal with it. 

I'm going to give you a rationale as to why we're coming from where 
we are, then give you an idea of the direction we're going, and then maybe 
later today I can get into more specifics. We gave a lot consideration to 
how we could approach this problem. The reason why we feel that it's imper- 
ative that we do is that as the state agency commits itself to providing 
assistance to school districts and in the development of our IRON—Illinois 
Research and Dissemination project, which is an NIE capacity building 
project—we 1 re literally beginning to develop files of resources that are of 
some magnitude. In the state of Illinois alone we're finding that there are 
a great deal of promising practices that are available and can be shared 
with the districts. If we're going to put funds behind that (and we are trying 
in every way we can to generate funds to support those activities) we make 
one demand upon the LEA's. (We're trying to very seriously to honor the 
integrity of the LEA's, to pass the responsibility dh to them for having a 
great deal of impact on the direction we do go.) We're requiring that they 
go through some logical , systematic planning process. We're saying to 
them, "You enter into a planning process, needs assessment, resource identi- 
fication, exploration of the alternatives, and we'll buy what comes but 
of the process." We're saying to them that as you go through that process 
and as you go through the identification of resources, identify and prioritize 
your problems arid come up with a number one problem that you would like to 
address. Then contact the IRDN. As a result of a search through the IRON 
they may come up with 14 alternatives, of which there may be three IVD 
projects. Maybe two or three projects are in the NDN but there may be 
another five or six resolutions to that problem or at least alternatives 

60 63 



to be considered. Our dilemma is, what kind of validation do we put those 
other promising practices to those other alternative resources through? 

Philosophically, I personnally give* some argument against some of the 
points that Bill has made. We're finding that it's difficult to operate 
in ah advocacy role, in a very close involved role, and be sincere with 
LEA'S if we feel we have to protect them, if we take the role of protecting 
them, becoming the consumer protection agency and saying to them, "You're 
not capable of deciding what projects are good, so we'll decide for you 
arid then we'll make it available for you." We're saying that v;e're going 
to involve them vri assisting us in developing what those values are. What 
is it that they value in a project? The other thing that we're saying 
is that we feel the IVD process is a pass or fail. We think part of the 
studies are saying that people aren't really adopting, they're adapting. 
If that's the case then there are parts of -projects that can be adapted, 
that are useful or of value, that are being eliminated because they have 
failed the validation test. Now I think I'm hearing that you're moving 
towards some flexibility around that — I'm glad to hear that, because I 
think it would be interesting to do an indepth study about whether it is 
adoption or adaptation that is occuring. It's been our experience that it 
is a great deal more adaptation than it is adoption. 

The other thing we're concerned about as we move with the IVD process 
in the manner that we have in Illinois, is that putting 39 projects on the 
diffusion network has become very expensi ve— extremely expensive. We use 
out-of-state evaluators in the process, arid the money that we're putting 
in there has to be questioned. Not that we're saying that it's not worth- 
while and valuable for the projects that we're putting through that kind 
of process. We can't affort to send visiting teams within the state, let 
alone bring some from outside the state. So what we're saying is that if 
we can identify some rank order, some instrumentation of what the values 
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are that the user wants in an evaluation process, so that you start at 
a fundamental level of the process and allow that project to work through 
those levels, there's no reason why a project will not reach an I VD level 
so a visiting team will come in. What we're saying is let's not eliminate 
them if they're not obviously a project that merits an onsi te visit. Of 
course, there's generalizing, there's heavy emphasis given to research and 
evaluation design that's going into some of these processes, and we're 
simply feeling that too many good valuable resources are bein\, eliminated 
because of that. 

Another point that is important to us, again dealing with getting 
the LEA's to share the responsibility for a state-wide dissemination 
effort, is that we don't feel the state agency should become the sole 
evaluator of promising practices. And we think through a mutual , coordinated, 
collaborative effort, we can get them deeply involved. 

We've tried something else that we've had a great deal of success with, 
cooperative consultants; For example, metrics is a big hot item in our 
state. We at the state agency were faced with a decision: do we hire five 
metrics specialists* put them on the state payroll, use them for three 
years and let them become obsolete, and then try to find something for them, 
or do we develop somehow the capacity to respond to 1 ,000 districts who 
must, by a certain date, do something in metrics? We came up with the idea 
of creating what we call a cooperative consultant cadre. We identified 
75 LEA math teachers and university math professors/instructors, brought 
them to the state office* and gave them three days of inservice on the 
policies and guidelines relative to the di recti on we wanted to go with 
metrics. We asked them to contribute five days of their time. We in 
return psy for their substitute, if it's required, arid pay their expenses 
to go to other districts. Our staff at the state agency then brokers 
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those 75 people into districts. For a multitude of reasons that's been 
extremely successful. There's notoriety for the people involved and it's 
an honor for a district to have their math teacher identified as a 
cooperating consultant. We've got a peer relationship activity going on, 
and we've broken down the credibility gap being a state official coming 
in and laying on policy. Our. role is to facilitate that effort and expand 
that to include more LEA involvement, and also work on the identification 
and validation ".of promising practices, to develop the instrumentation, the 
process to enable that to occur. 

To summarize, given the commitment we have made to working with the 
districts, we have to have a more comprehensive model, we've got to go 
beyond IVD and JDRP. I don't want to be perceived as negative to the IVD 
or JDRP. It's not that at all, we're just feeling that it falls short of 
what we want to do, what's being dictated to us by funds, what's being 
dictated to us relative to our overall program. 
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PRESENTATION BY DR. JOHN OSBORNE 

Michigan has been actively involved with the concept of validation 
since 1973. We have participated in the- national IVD process; we have 
people who are trained in that process; and we've had projects that have 
been classified by the JDRP. We have not had projects in the last two 
or three years that have been nominated for IVD by the Michigan Department 
of Education. There have been some reasons for that. We found that 
every time we nominated a local project for national validation it was 
costing between $1,000 and $2,000 to pay for the validation and then it 
might not get classified. We found that local politics sometimes was a 
problem. We had one experience where the validation procedure was quite 
embarrassing to all parties involved because of local politics. To go 
through the process of IVD and be nominated to the JDRP — assuming that 
you get by the JDRP--doesn 1 t mean any funds will be available and a lot 
of our school districts were resenting the fact that they got through 
that process, got a certificate, and there were no funds to respond to all 
those questions and calls for materials that were coming in from throughout 
the country. Then we had projects that were going through the ?DRP that 
eventually got funded but the level was very small. In terms of our 
salaries in Michigan a $30,000 grant may pay for 3/4 of the time for an 
individual to work on the national scene. So for a lot of reasons school 
districts in Michigan opted not to participate in the national process. 

As the same time that we were involved with the IVD process, bur own 
state board of education directed our department to come up with a proce- 
dure to do the same or a similar thing as the IVD and the JDRP. One of 
the first things which the state board of education did was to develop a 
Classification Committee. Now this committee is similar at the state level 
to the JDRP. It's headed by our deputy superintendent, and its members 
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include the head of research and evaluation, the director for planning; 
the associate superintendent for elementary and secondary education, the 
associate superintendent for extended learning (because we get involved 
with professional development) , and the assistant superintendent for 
school arid community affairs. 

One of the charges made to the Classification Committee was to seek 
successful programs throughout the various funding agencies in our department, 
in other words, not to focus oh Title IV-C or Title III, but to begin 
getting Title I projects surfaced as successful practices to be promoted by 
the state board of education. Our proeess is actively pursued by the state 
board and the staff of the department of education. We are in an advocacy 
role. We are advocating those things that are successful. We want to bring 
that to the attention of school people throughout Michigan and to do things 
systematically that will encourage school districts not to adopt, not to 
adapt, bjt to install. Now to you, "installation" may be a weasel word but 
I'm tired of defending "adoption" and "adaptation." I think "installation" 
lets anybody put the context they want to that word. 

We have in our bank of programs Title IV-C projects, Title I projects, 
programs from our own state process called Chapter III, which is simi lar to 
Title I but with a different client group, we have one or two from vocational 
education, about six from special education, two from career education, and 
we're in ths process of nominating five from the area of professional 
development. So our efforts have been to go department-wide and not focus 
only on Title IV-C or the old Title III. 
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Earlier I mentioned cost. We can have a validator ebme into any of 
bur projects and it will cost about $125 for a day arid a half of work. 
We add appropriate department staff to the validation team. If it's a 
math project, a math consultant goes along and verifies that it's a good 
math approach and so forth. The team leader of that validation visit is 
an evaluator. Again, we're trying to demonstrate throughout the state that 
our student data show our programs are successful. In other words, they 
do make a difference in the achievement level of youngsters. 

Some of you were asking questions earlier of other presehtatbrs about 
how the JDRP evaluates those reports, what criteria are used, and what 
procedure is used. I would respond by saying that Michigan's procedures 
is similar and I do have hand outs to pass out to you and I'll walk you 
through it very quickly. (See page 71) 

We've only nominated projects for classification that have been state 
or federally funded up to this year. The Classification Committee has come 
up with four levels or categories that a project or program goes through. 
One is the planning mode, that's the P, one is the developmental mode, that' 
the D, and one is the experimental or replication mode, that's the E. The 
second D is the demonstration and the I is the installation where we 
demonstrate and have school districts install. We have criteria for whether 
a district has reached the completion of the planning phase. 

After a project has gone through a developmental mode, be it a Title I, 
Chapter III, vocational education, special education, or whatever, a team 
is sent out to do an onsite visit utilizing a form to determine whether the 
projeet has merit for being selected to become a demonstration site for the 
entire state of Michigan.' We're not saying it's not a promising program 
at that point. That part of the state may indeed have use for that, but the 
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Classification Committee is advocating programs from a state-wide viewpoint. 
We're attempting to address state-wide needs as they appear at the present 
time. If a project is judged to have met the developmental criteria 
successfully, it is nominated for classification, the committee examines 
it— blesses it so to speak—and it says, "You are now an experimental site. n 
In the developmental mode, a project does have an experimental design. All 
of our projects have experimental designs in that developmental mode. And 
when we make a determination that indeed we want to nominate it, they've 
■r- ready proven to themselves and to our staff in the department that they're 
successful in raising cr effecting student achievement in a positive manner. 

In the experimental moce, we give a school district resources to go 
into a different district to replicate that program. And this is where we 
differ with the federal approach in some respects. We look at export abi lity 
or replicabi 1 ity not in the sense that the project can be picked up and 
used by another school district, or that it can be cost effective, because 
school districts will make their own determination about that. What we're 
interested in as a state education agency is whether or not the results are 
exportable. And we will not take anyone's word for that. That is why we 
go into a different district. Let's say we have a project in Austin and at 
the end of the developmental mode it has been judged as successful. We 
would give San Antonio a period of time, normally at least one year, to 
implement that program and see if they have comparable results. Not 
exact, but comparable. And during that phase, the original developer site 
is acquiring experience in how to conduct i n^ervice training— and that's a 
question always asked but is seldom demonstrated in most IVD approaches - 9 
state or national --but we're going to have evidence after a year that 
indeed a school district can provide inservice training. During that year, 
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they will have the opportunity to pull together all of their packaging; 
We're giving them the resources to package the training module so that 
everything is ready. 

After the experimental phase is completed, we then do the on site 
validation arid verify student data, comparing it to the original site. Do 
they have comparable results? If the answer is yes they are nominated for 
demonstration status. 

if a project makes it through this whole process and becomes a demon- 
stration c1 "te, then our department provides funding averaging about $75,000 
per year for a minimum of two years to demonstrate state-wide, provide 
inservice training, and to provide follow-up to anyone who installs that 
program. In addition, we have $5,000 grants for school districts to install 
a demonstration project. 

In terms of the Promising Educational Practices, we may be a little 
further than Illinois although we haven't put those PEP's in a package. 
We do have 10 regional supplemental centers in our state that are being 
funded with supplemental funds to identify on a regional basis promising 
educational practices. One of our centers has come up with a folder as 
one approach to identifying PEPs. 

Every single teacher in that county (intermediate school district) 
gets this folder. In it are recommendations for sharing ideas, sharing the 
program, and information on the nomination procedure. Thero is a reviev. 
team mentioned in the folder which communicates from the teacher back to 
the intermediate school district or the regional center. The idea is to 
develop communication between the teacher and the center people in order 
to begin sharing classroom ideas. From that we'll begin to ideritify 
promising educational programs and practices. 
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Maybe they heed this form: "I need help in getting an idea going." 
This is being administered by one of our centers that is very much into the 
marketing approach. They have these things on display at all the malls, 
they have radio spots, billboards, and so forth throughout the county. 
Another county, a regional supplemental center, talks about what a promising 
program is and then they have a process by which local districts can nominate 
a program. They have site visitation forms. In other words, in that county 
the curriculum people, the assistant superintendents, and directors of 
instruction will form a team with a professional evaluator from one of the 
school districts and they go out at their own expense and conduct an orisite 
verification-validation process and begin identifying on a county wide 
basis those programs that are promising to their region. Our intent then 
is to take all the programs, and we've identified 300 as of today, begin 
screening them out, and come up with which of those 300 are programs that 
have application to all the school districts in the state, not just on a 
regional basis but throughout the state. We would then encourage them to 
apply for classification and we would provide funding if they are anywhere 
on that continuum. Maybe they are just at the planning stage. They already 
have an idea developed but they may not have the necessary evaluation data 
to be classified. We would provide fiscal support to those dv ' icts to 
acquire the data to be classified and be nominated as a demonstration site. 
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Michigan's Categories of Projects 
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PRESENTATION BY DR. PRESTON C. KR0NK0SKY 



There are a number of things you should consider today, tomorrow, and during 
the next weeks and months as you work through and continue to refine your state's 
validation process. First* since you are associated with a state education agency, 
there is a book that is going to cause you lots of problems. The book is called 
The Literacy Hoax . Its subt.cle is, The Decline of Reading, Writing and Learning 
in the Public Schools and What we can-Do-Ab outfit . Its author is appearing on 
many of the national TV "talk" shows. He claims that he has the solution to 
today's problems of American education. That means he's got the solution many of 
the people in your legislature are going to adopt, perhaps adapt, and that's 
going to cause you and state education agencies and everybody associated with 
public education in the country, lots of headaches in the next weeks * months * and 
years. 

The thing that is so dangerous about this book is that its author claims to 
have identified all the problems that all the experts with all their PhD's have 
not been able to identify. Further, he has come up with very simple, down to 
earth, practical solutions that can be put into effect tomorrow. And in the 
process save millions and billions of taxpayers' dollars while raising mathematics 
and reading and communication scores dramatically. If you haven't read it, this 
is something that you are going to have to deal with and I urge you at your 
earliest convenience to read it. 

Second, I'd like to express publicly my appreciation to Linda Reed of the RSD 
Interpretation Service at CEMREL in St. Louis for her excellent synthesis concerning 
validation. I believe everyone received a copy of it last night and I know you 
stayed in your room last night ?nd read it. It is an excellent contribution; I 
know Linda Reed would appreciate receiving ybUr constructive comments and feedback. 
It's another example of how within the Research and Development Exchange a couple 
of the contractors from time to time are able to cooperate. Further, we appreciate 
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Sharon Koenigs and Dane Ranis of CEMREL's Regional Exchange making a number of 
our presenters today available to us. That's another form of collaboration and 
support. 

Third, I'd like to alert you to another paper. As you read Linda Reed's 
synthesis you will note that she makes reference to the Rand Study— all eight 
volumes— but probably the one most of us will read thoroughly is Volume 8 since 
it's a synthesis of the earlier seven volumes. And since the entire eight 
volumes constitute several thousands of pages most of us won't take the time or 
the trouble to n__d all of them. That also means a lot of other people are only 
going to read Volume 8 and unfortunately many people will simply adopt the 
conclusions that Volume 8 advances as being totally accurate. 

I believe that there are other points of view that can be drawn from the 
data presented in the first seven volumes and that there are other points of view 
that could have been expressed in a summary volume such as Volume 8. One such 
point of view is that expressed by Lois-ell in Datta of the National Institute of 
Education. I do not have her paper in final form, I have only the version which 
existed as of December 28, 1978. Its title is: "Damn the Experts and Full Speed 
Ahead: An Examination of the Gases Against Directed Development and for Local 
Problem Solving." So, if you're involved in evaluation or validation in any way, 
I think you'll be very interested in reading Dr. Datta 's summary and interpretation 
of the Rand Study, which does differ from that presented in Rand's Volume 8. If 
you allow people to accept the position expressed in Volume 8 as being totally 
accurate, a lot of you, make that "us," will be in unemployment lines very soon. 
When you add that document to the previous document, T4i e Litera cy Hoax , I hope 
we're all intelligent enough to fill out the unemployment claim forms. 
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Fourth, as a student of the change process and a student of dissemination 
arid diffusion* etc. * I'm assuming that you have had the opportunity to read 
excellent syntheses of some of the various aspects that all of us concerned with 
evaluation and validation must be aware of as we grapple with this question of 
implementation. Increasingly we have evidence that it is wrong to draw conclusions 
about the effectiveness of a treatment, i.e., the effectiveness of an innovation, 
until we have evidence as to whether or not that innovation has in fact been 
implemented and, if so, to what degree. Perhaps there is validity to the argue- 
ment that no innovation can be repl icated 100% in its second or third or fourth 
repl ication site. Some experts say the best you can hope for is about 80%. 

In any event, in the R^v4^ew^3f Educational Research that on the cover says 
Spring, Volume 47 issue, is a long article by Michael Full an and Al an Pomfret , 
"Research on Curriculum and Instruction Implementation. " I think it's one of the 
best current statements about what is happening in trying to measure implementation. 
If you have not had a chance to read it, I urge you to read it before you go too 
far in trying to devel op a val idation process that doesn't take into account soine 
measure of implementation. 

The article reviews a number of ways to view implementation. Not the least 
of which happens to be that pioneered by Drs . Gene Hal 1 and Susan Loucks at The 
University of Texas Research and Devel opment Center for Teacher Education. Those 
of you who have been working with the SEDL Regional Exchange for some time know 
that we've been trying to acquaint you with Gene and Sue's work. I'll be the 
first to say though, don't put all your bets on their approach either. I suggest 
that you read as widely as you can from a variety of sources as to different 
methods of how you can look at implementation . But please don't make the mistake 
that we've been making for the last 10 years and assume that you can draw a 
conclusion about the effectiveness of innovation simply by looking at pupi 1 change 
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scores without knowing whether the treatment has in fact been implemented. 

Last but by no means least * one of the current catch words, fads, in-things, 
whatever you want to call it, is ethnography.. In fact, one wag recently said 
that the real name for N IE is the National Institute of Ethnography, because If 
you put into your proposal title the word "ethnography" you have a much better 
chance of having the proposal funded than if you leave that word out. (Remember 
about 10 or 12 years ago, if you put "innovative" in your proposal title how that 
increased your chances of getting funded? Well the word to use now is ethnography.) 

There is a lot of argument concerning whether you can use ethnographic 
techniques as a form of evaluation. I am probably the only person in the world, 
and I'm sure the only person in this room, who'll admit that he's not an expert 
on ethnography. I can't even spell the word without looking at it. In fact, I 
don't think I even knew it existed until I picked up this issue of Review of 
Educational Research . So if your knowledge base concerning ethnography is as 
limited as mine, I urge you to get this issue of the Review of Educational Research 
because it has an excellent article (excellent because I know nothing about ethno- 
graphy and therefore anything is an improvement over my current knowledge base} 
entitled, "The Use of Ethnographic Techniques in Educational Research." 

You may want to consider this article as your introduction to the possibility 
of ethnographic techniques being valid indicators of evidence of program effective- 
ness. Maybe it doesn't always have to be the classic pre/post, grade equivalent, 
change scores, etc., --you may want to consider this instead. Though I understand 
from talking to some of the "experts," there is a lot of argument as to whether 
you can use this approach as a valid evaluation technique in implementation research. 

Now to the main point. Our laboratory has served as a sponsor of a Follow 
Through model for 11 years. As you well know, Follow Through is the most, documented 
research and development effort the United States of America has ever engaged in. 
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It's also the most expensive. As of this time, the U.S. Off ' ^ , ^uu^atiori has 
expended in excess of $500 million on the National Fo Hot' '.V* . vi.gh Program. Fifty 
million dollars of that total amount has been spent on the evaluation of the 
National Follow Through Program. 

If you read certain newspaper headlines in the summer of 1977, those headlines 
said, "Follow Through a Failure." Unfortunately, there were several other 
paragraphs, but most people didn't get past the lead paragraphs (at least not 
certain editorial writers). Many persons preferred to say that the U.S. Government 
spent $500 million and it all went down the drain; What the official USOE press 
release tried to say was that about 85% of the 22 educational models that were 
involved in the program either produced no significant difference or unfortunately 
had a negative effect upon the achievement of the pup Ms involved; Later paragraphs 
said that three of the models, or about 15%* did show positive significant changes 
in pupils' achievement. 

There is another very important point. All of the materials, at least all of 
those used in SEDL's Follow Through Model, which happens to be one of the three 
successful models, had gone through a very rigorous, every expensive, very time- 
consuming process of design , development, evaluation, and revision, for validation. 
We knew when we first began to put those materials into our Follow Through Program 
classrooms that the materials "worked." They "worked" in our design-test class- 
rooms, they "worked" in our pilot-test classrooms, and they "worked" in our field- 
test classrooms. We had a rigorous evaluation design and "hard" evaluation Jcta 
that confirmed that. The strange thing was that when you put those same materials 
into 200 classrooms, spread over 19 elementary schools in seven school districts 
in four states, they varied in their pupil-effect across the 200 classrooms. 
Theso were the same materials we had previously developed. We knew they "worked," 
they were excellent, the best available anywhere. Why didn't we get uniformly 
positive results? 
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Perhaps the foregoing is why the U.S. Office of Education several years ago 
decided that its efforts in the late 60's and early 70's, the so-called Panel 
Review of Products (PROP) (some of you were associated with that), was not the way 
to tell the educational consumer whether a particular innovative instructional 
material was worth their time and money and the involvement of their children. 
It takos more than just excellent materials or products. It takes products 
successfully implemented in a classroom situation. 

Unfortunately, as Follow Through demonstrates, excellent products in brie 
classroom produce one set of pupil effects, while the same excellent products in 
another classroom produce an entirely different set of pupil effects. Something 
in the classroom situation causes or permits the differences to occur. The 
differences may be "caused 11 by the teacher, the principal, the pupils, or something 
else for all we know. But, it is obvious that you cannot take products that have 
gone through an elaborate RSD process and guarantee that they're going to work 
everywhere. So what you have been about for a number of years is something we do 
need to continue to be about: we must validate programs and we must try to 
identify exemplary practices. 

A member of our SEDL staff has been given the responsibility of working with 
SEDL's seven Follow Through project sites and trying to prepare them and their 
written presentations for submission to the 0E/NIE Joint Dissemination Review 
Panel (JDRP) . Jack Lumbley is one of the few people in the country who has had 
the "pri vilege" of working with as many as five project sites to go before the 
JDRP. The five sites range in size and complexity from San Diego, Texas to the 
Los Angeles Unified School District, the second largest school district in the 
country, and a number of school districts in between. 

We have seven sites, and by the end of this summer we hope to have taken six 
before the JDRP. Three passed their very first time. The fourth failed by one 
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xrot™ arid we are ready to submit it for a second time. We are ready to submit 
(or resubmit) the fifth one in the next week or two* and as Jack has the time in 
the next couple months, we 1 11 be ready with Philadelphia, another very large, 
complex school district. I personally believe that by August of this year, we 
will have six of our seven sites validated by the JDRP , whatever that means, which 
I think is a very good record. But what we're able to demonstrate is essentially 
that the same products produee different results with si mi liar sets of children 
in different classrooms. That is something your validation process must be 
prepared to address. 

Good luck to you . If we at SEDL/RX can be of assistance to you in the next 
two days or two years, please do not hesitate to give us the opportunity to work 
with you. Thank you. 
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PRESENTATION BY MR. JACK LUMBLEY 

On several occasions I have had the privilege or distinction of having 
sat before those persons who compose the JDRP panel at any one time. It's 
always extremely interesting to say the least; There are presently 22 
persons who constitute the overall membership for the JDRP. Generally, at 
any or.o time, seven persons generally show up because that's all they have 
to have in order to convene a presentation session. I have a present 
listing of all of those 22 persons rom OE and NIE and I'll be happy to 
provide any of you Who are interested with this. 

The present chairperson of the JDRP comes from NIE, John Evans. 
From the experience that I've had before the group, the chairperson is an 
extremely influential person in terms of his or her conduct of the session 
and in terms ultimately, of one's success or failure. The reason for that 
is at the beginning of eaeh session the chairperson has the responsibility 
for a brief description of the actual presentation itself. That description 
is often times the first that the other persons on the panel have even 
heard. Some of them may come completely unprepared and depend comnletely 
upon the synopsis provided by the chairperson. Because of that* it's often 
the ease that the chairperson has complete control of the discussions, the 
question ing, and so on and can often sway the ultimate outcome. Perhaps 
that's an extremely biased view-- 1 guess I ought to say that—but it seems 
to me that is sometimes the case. 

As Preston indicated, the e-nerience that I've had personally has been 
with the school districts themselves, with the LEA's who have been installing 
our Follow Through approach. Also, those school districts drastically 
differ in size, resources, and sophistication. I specifically point that 
out to give you an indication of the fact that the dobs that I have done 
with LEA's have almost always been after the fact, that is* whatever 1 data* 
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whatever evidence there is of program success, are already collected and 
already exist by the time that I arrive and it's my job to take what- 
ever there is and to try to place it in a sensible form which will sway 
this body of persons to approve the project. 

That fact stands in opposition to some of the statements or suggestions 
which appear in this document which you all received from CEMREL. It is 
suggested in there that one should begin planning to approach the JDRP 
almost at the same time that you actually begin your project, that is you 
have to begin thinking about a plausible design for evaluation from the 
outset and be prepared to do whatever is required to obtain th .-se sorts 
of data which presumably would be successful in convincing the panel. Well 
in these cases, that has not happened because the designs were already 
firmly entrenched before I appeared on the scene, and I took whatever they 
had and shaped it up. There are some common pitfalls also stated within 
the same paper, which "perhaps you would find interesting if y ''11 look 
over them. They are pit-falls in the sense of some rather nuiidn sense 
suggestions as to things which ought to be considered and avoided in pre- 
paring the presentation and actually defending the presentation before the 
panel . 

The JDRP is interesting and is distinctive of the other processes of 
validation which are presently around in that it is completely data based. 
The actual presentation has to take the form of a soundly developed 
experimentation paradigm. There an* hi- opportunities for any site ' 'sits, 
obviously. Whatever the panelists decide will be determined exclr. ively 
by whatever it is you are able to cram into those 10 pages that you are 
allowed for your presentation. So things have to be v*ry thoughtfully 
constructed, thoughtfully defended, because that's your only opportunity- 
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Panelists generally are concerned with things Such as statistical signifi- 
cance, which I suppose you all are acquainted with. They're a ;i>b concerned 
v. * 4 h something they call "educational significance," which is something I 
v.. -lever taught about in my graduate statistics courses. Nobody ever 
discussed educational significance; I suppose it wasn't of any concern 
But the people on the JDRP do express interest in that. And it's up to 
each presenter to devise s rifle way of converting statistical evidence to 
some statement concerning educational significance. There isn't a set 
pattern 'or doing that; you have to be able to do that yourself somehow. 
Generally the evidence takes the form of student outcome related data, 
John, [John Osbbrr>J something that you said today in your presentation 
was interesting to me because you indicated that your state had opted to 
bypass the JDRP because a? your interest specifically in child outcome data; 
That is principally the kind of evidence that this panel finds especially 
convincing. 

There is also an obvious concern with the cxlent that presenters can 
indicate some degree of transferability or general inability of the product, 
:he p : gra^, from i us initial implementation to some other point in the 
future. The?e people sit on this panel somehow pass a judgment on 
products/practices/programs and it is ndeed the case that if they approve 
you, that causes you to be eligible, sometimes- tg v ^ additional federal fun ^ 
to spread the word about your projee./ program/ practice; Because ;f t at 
there is an obvious concern for sJma ability to -ip^enstrate tha extent thoc 
you believe t^at your ;?roj t c + *l ^ transferred some place else, can 
expect to accrue the same results to the participants that it is exposed 
to as it first did. So you ha 'e to be cuU; to address that concern as well and 
sometimes tnat is not easy; 
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Specifically, in terms of the structure of a presentation* it's 
always interesting to me to deal with a new school district because it's 
the same material, the same implementation* across all sites* but the 
effects are not the same. So obviously, that has the implication of 
influencing your specific set of claims of effectiveness for this project/ 
program. The procedure thut I try to pursue—and it has paid off in the 
past—is to spend a considerable amount of time examining whatever data 
are presently available and try to devise some conceptual whole from that 
corpus of data and specifically to come up with the strongest set of 
claims of effectiveness that I can with the caveat in the back of my head 
that I have to have some evidence to specifically back up every single 
claim that I'm interested in proposing. \n thai senr-e; each of the presen- 
tations is completely distinctive. It': <r * .^-irig to me to start from 
scratch and to try to devise a unique set of clain.i for each of the c * 
presentations, which 1 hopevully can back up with whatever evidence would 
be required for them. T say that simply as an indication of, almost as a 
refutation of, the statement I often times have seen that purports to say that 
there is some single way of preparing a presentation. That isn't the ease. 
You actual ly have a wide degree of freedom in constructing a presentation. 
7I.£ ^ n ly concern is that you construct claims of effectiveness in such a 
v/ay iWzi you have whatever evidence is requi red to back them Up. As I 
suppose all of you are aware, tM?re are some rather obvious advantages to 
going rugh this trial by fire because approval places you within the 
NDN, *t puts a description of your prograrn/project/practice/produet, what- 
ever, into the NDN catalog which receives wide distribution across the 
count ry; and also it qualifies you, theoretical ly, for additional funds 
from the USut to spend in spreading the word about your validated program: 
So those are tile advantages. There are the obvic.is disadvantages as well. 
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But depending upon your own conceptions, I think the advantages outweigh 
the disadvantages. 
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PANEL DISCUSSION SYNTHESIS 



The panel discussion, cm the morning of the second day of the conference* 
was moderated by Ben Harris: Panel members were Christine Dwyer, William Hinze, 
Preston Kronkosky , Jack Lumbley, Bad Srossner, and John Osborne; each of whom had 
made individual presentations to participants the day before. The discussion 
covered the following topics. 
The- Teac her 

The teacher was not often mentioned, if at. all, in state team discussions. 
It is important to keep in mind the teacher* or the classroom as the unit for 
validation and not focus exclusively on administrative concerns Because it is the 
teacher Who must: 

• ' low the essential elements of the innovation; 

• understand why all of those elements are important; 

• understand the purpose, use, and value of evidences of effectiveness 
end t>e teacher's role in collecting the evidence; and 

• ostprsrhie the feasibility of the program in the classroom: 
When to Validat e? 

There must be a distinction made between the var^Lis str-ges during the life 
of ci program; the development, the validation* and the dissemination stages. Too 
often, the distinction is lost and pressure to validate a program still in the 
developmental stage results in completing neither of the first two stages satisfac- 
torily. F"i*\: k and Peterson, among othrs, have concluded that the implementation 
process ta':es three to five years* and only at that time can one begin to collect 
meaningful evaluation data. Jne strength of Michigan 's ~,tate program is that it 
classifies appropriate programs as experimental. Arkansa: ?iso includes a stage 
during wfv'c^ the project completes its developmental activities, in this case within 
a year. Arkansas , like Michigan, requires that a program not be replicated until 
it is validated. 
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Who Should Vali date? 

In connection with the stages of a program-, the question arose whether only 
a second generation program should be validated arid/or disseminated. Second 
generation program directors, while possibly not aware of the details of a program's 
creation, may be more objective about program effects than the developers. Second 
generation sites (that is, adopters) may have a clearer insight into implementa- 
tion problems and needs, and know more riSoiit matching a school's needs with an 
already developed program rather than a developing one. Potential adopters may 
value those insights as much as program details. And* of course, it is those 
second generation sites that "prove" whether the program can be transported, and 
therefore validated, and uiere^ore disseminated! 
Paper Fi-le- o r Program Fi 4^ ? 

The temptation to validate a program based oi: )y on developer-writer descriptions 
plus examination of program products should be resisted. Ask whether the program* 
the process, or the product is to be validated. The validation questions which 
need to be answered will differ for each category, and the usefulness of the 
validation data will differ accordingly. Paper fi les— col lections of reports, 
brochures, teaching materials, an 1 the "h'KG--wi11 testify to procedures and processes, 
in general, and will indicate the quality of the teaching materials. What will not 
be clear is the actual operation of the program in a school, and its ejects on 
teachers, students, and school routine. Paper files will not show wh< ler imple- 
mentation is occurring in all classrooms, or whether the manner of implementation 
differs signif icantly across cl :ssrooms. So-called program files, on the other 
hand, address these questions because they will provide for on-site visits either 
through a visiiilng validation team ciuri.ig the validc n process, or by the 
establir'ime*- 1 of demonstration sites during the dissemination stage. Obviously, 
staffir d budget limitations on validating activities must be considered, but 
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the most productive use of available funds might be to begin a pilot-test of 
the on-site visit to determine how useful it is to .he purpose of the validation 
procedure: The state may certainly trust an LEA's description, but will not 
really know a program until it se^ it in action. Paper descriptions are simply 
1 imi ting. 
Why Validate? 

That there are separate sets of questions which must be asked depending bri 
whether a process or product is being validated leads to current implementation 
theory, which states that, because people, places, and situations differ, there 
is not pure adopti r. , there is only adaptation of a program to someone else ' s 
needs. Wh^t then are the implications for diffusion? What conditions are 
necessary to reproduce a proqram in as nearly the same state as it was originally 
validated? Is this possible? Is it possible to describe accurately that "original 
state" er *«- validation merely the approval of one frozen moment? Even 

first c , rogr?^s change. If everything is relative, is validation 

itself va'. If there appear to be more questions than answers, and if the 
questions are difficult to answer, it is because the panel and participants, in 
the end, asked the questions that now have ho answers. As ways of determining 
essential characteristics of innovations are developed* and then ways of assessing 
their implementation in the classroom emerge, perhaps some of these questions will 
be answered. For now, the actions reflect the current validation dilemma. 
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SUMMARY OF STATES * CONFERENCE WORK* 



Int: validation conference was p.lanned to provide each state with the 

time to work as a unit toward its special goals iri validation. Each team based 
its work on needs stated during the planning stage and on relevant information from 
conference presentations. On April 19* teams met together from 1:30-4 :30pm and 
on April 20; the teams worked from 10:20-1 M 1:00am, The following reports summarize 
the work of each state's team. 
Arkansas 

At chc time of the conference, Arkansas had a validation procedure based on the 
IVD process. Programs are rated in one of four ways: approved by the JDRP; approved 
by the IVD and moving into the JDRP process; completing the developmental stage and 
moving into the national validation process; and showing merit but not judged on 
potential for validation. LEA's may consider projects in any category except the 
third. The state team worked with SEDL/RX evaluator Jack Lumbley and was most 
concerned wi t!i revisions within the currrn framework, specifically with suggestions 
for increasing the state 1 ? success with tiie ^ :P process. The team was interested 
in and willing to share its process and JDRP experience with other state team 
members : 
Louisiana 

At the time of the conference, Louisiana had no validation procedure in effect, 
but did have a commitment by the Dissemination Task Force of its Special Purpose 
grant to construct one. The team was interested in funding procedures, housing 
and staffing, and composition of on-site visit team- Its orientation was toward 
state validation procedures. 



*The Mississippi SEA sent one representative to the validation conference who sat 
in with several qroups as a participant-observer. The lississippi SEA jdiiierf the 
SEDL/RX service ^rea in April and so was not involved Iri the planning p-^cess 
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At the conference, the state team worked with Bud Grossner, Manager of LEA 
services with the Illinois State Department of Education, and conference resource 
person. The team was interested in details or the Illinois plan: problems in the 



first stages, education agencies concerns , establishing a service that people 
would be aware of and would use , incorporation of university personnel , self- 
assessment, and Illinois' multi-point scale for ranking programs according to 
the value of their various components, 
The team eventually decided that: 

• consideration should be given to developing a state process 
rather than adopting the IVD process; 

• there* is a heed to develop and maintain a file on educational 
programs and that the validation process will allow. potential 
users to assess their applicability by using specific evaluative 
cri teri a: 

• the program file needed to be expanded from NDN programs to 
state validated programs and that the file should also include 
practices, processes, and products; 

• consideration must also be given to criteria for validation, 
training project directors and evaluators to study projects 
throughout the state to determine their appropriathess for the 
Program File, and subsidizing local school systems for disseminating 
their projects after validation. 

N ew-Me^i-c a 

At the time of the conference, New Mexico had no state validation process, 
but did have a second draft of a proposed process ready. The team 1 s primary concern 
was to get assistance in reviewing the draft and to get suggestions for implementing 
the procedures. 

v the conference , the team worked with resource person Christine Dwyer, from 
RHC Research Corporation. Discussion include: 

• criteria for validation: 

• the length of the proposed application form; 

° the possihility of associating the validation process with the v 7EA's 
accreditation cycle* 

• cr<* need to" a s in:p ••a"! id?* >,t\ rrodel ; 
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• the difficulty of including on-site visits when funding is limited; 

• elimination of commercial products from cons; deration for validation and 
the need for guidelines establishing at what point a program is defined as 
commercial ; 

• what makes * nomination formal fee wive? Can such a for *ct as a 
screening oovke before the final apol ication form; 

• by who:. *r • v.tieh will a program be nominated? 

• what are the implications for va I i cation when a program appeals to have, 
achieved its goals, but its contend is innaccurate or contains race/sex/ 
age/handicap biases? 

In addition to discussion, the team received Dwyer 1 s specific observations 

on the proposed validation process and worked with her on additional revisions. 

The team also requested that Jack Lumbley , SEDL/RX evaluator, go over its final 

draft and provide additional comments arid guidance. A mid-to-late summer meeting 

was tentatively scheduled for completion of the final draft for Lumbley's review. 

Okl ahoma 

Oklahoma, at the time of the conference, ha* a statewide process and used IVD 
forms though its process was not formalized. Team members were interested in 
specifics* such as: 

• reducing length of forms; 

• methods for handling programs which appear :o be effective but which have 
problems with statistical analysis; 

• length of site visits; 

• size of on-site teams; 

• improving .feedback to state about areas of improvement for state programs 
which do not receive JDRP approval; 

the need for a system to allow reports to be compared without having 
identical validation procedures. 

Trie tea* worked with resource person Dr. William Hirize* of the U.S. Office of 

Education 'r Division of Replication, and began working on a written description of 

the "ocedur- currently followed, the details of whi c:^ could later be amended and 

fo nalized. 
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At the time of the conference, Texas had a statewide validation procedure 
for i rid i vidualized ins;. ruction techniques and was contemplating an expansion of the 
procedure beyonc j;-,a t content areas, especially guidance and counseling, special 
education, arid staff development. The team came to the conference with such 
questions as, what purposes are there for validation? Should the expanded procedure 
build in the grooming of the program for national validation? What kind of emphasis 
on replication makes sense when there are no funds for statewide model programs? 

At the cor. . erence, the Texas team worked with resource person John Osborne, 

Supervisor of Experimental and Demonstration Programs for the Michigan State 

Departrcant of Education; the team concentrated on writing recommendations about the 

proposed change from individualized instruction to content areas to present to the 

Advisory Committee of the Demonstration Schools Network on May Si, 1979: Several 

issues and questions arose: 

• if the TEA moves from individualized instruction to content areas, 
will it have to change appl ication forms to account for <J • Cerent 
kinds of evaluation data? (Most team members said "yes..") 



Should there be separate criteria for exemplary program i_r« s<\o:rure 
content areas? (Most team members we re uncertain and borrowed : ibh/RX 
information on Alaska's separate criteria to review.) 

As the number of exemplary programs grows, the yearly task of reviewing 
them becomes increasingly difficult to manage. What solutions are 
avai lable? 

Should the present system of validating for one year only oe changed 
to every two .year?? (Met team merroers were undecided.) 

Staff development prdsrSms shouid b> stressed, but wha^ evidenca(s) 
of effectiveness can be required? iVre demonstration sites for staff 
development for guidance/counsel; g programs possible? Practical? 



BeLausri Mc copies of demonstration site products o~ brochures are 
kept at TEA, interested people are referred to a contact person at 
the site. Collection of products ard brochures was recommended. 

More assistance with promotion and dissemination should b£ offered 
to demonstration sites, if time and moiiey perrrli L* 
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Before final validation approval is given, dissemination materials should 
be submitted to TEA to complete the application; 
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LIST OF PARTICIPANT PACKET "CNTENTS 



Each participant at the? Regional Validation Conference recieved a packet of 
the following information: 

• agenda; 

• objectives; 

• expected outcomes; 

" bibliography of resources on validation from the SEDL/RX Resource Center; 

• nne copy of Issue in Dissemination: Practitioners Access to Knowledge 
Base , by Ann E; Matthews, northwest Regional Exchange; 

• chart of resource people and SEDL/RX staff ^signed to each state team 
group; 

• information sheet on the 5EDL/RX; 

• information sheet on SEDL; 

• map of immediate conference vicinity, with nu.ne and location of suggested 
restaurants tar lunch; 

• reprint from a local city guide listing restaurants (by style of food 
served), movies, recreation centers, and tourist attractions; 

• paper and pencil . 
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MATERIAL RESOURCES 

The validation conference provided an opportunity to make available a variety 
of resources. The following pages list: 

• sample resources for validation; 

° resources on validation from the SEDL/RX Resource Center; 

• resources distributed by conference presenters. 
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SAMPLE RESOURCES FOR VALIDATION 



The SEDL/Regional Exchange will have several material . resources avai 1 able for 
your use during the Validation Conference. Many of them Will be available for loan 
after the conference, _ i f you desire. Below is a.sample of the materials which will 
be available. A complete list will be included in your conference packet when you 
arrive Thursday morning. 

# Pathways to Success: A Resource Manual for the Dissemination of Successful 

Educational Programs. Dorothy Soper. New Jersey State Department of Education. 
Trenton, New Jersey: September 1974. 

The manual describes the New Jersey state plan for dissemination of educational 
programs. It includes the theoretical rationale as well as descriptions of 
the validation procedure and dissemination requirements. Validation in New 
Jersey requires a commitment by developers to dissemination, and dissemination 
efforts of validated programs are supported by Title IV-C. In addition, 
programs must be certi f ied as cost-effective and their maintenance must not 
increase a consumer district's per-pupi 1 expenditure. Also available will be 
New Jersey ' s catalogue of demonstration sites . 



• The Joint Dissemination Review Panel IDEABOOK. G. Kasten Tallmadge. U.S. HEW: 
NIE and QE. Washington: October, 1977. 

The Ideabook is specific to the operation of the Joint Dissemination Review 
Panel and has two purposes: 

1 . "To illustrate the many ways to gather convincing evidence 
of the effectiveness of educational innovations." 



2. "To suggest how to bring current evidence of effectiveness 
together in a succi net and forceful manner. " 

Included are many examples and models pertinent to evaluation techniques. 



• Packaging Your Educational Program, Draft Version. Fred S. Rosenau and 

Diane H. Mclntyre. BE: Far West Laboratory. San Francisco: January, 1977. 

"This handbook for educators focuses on the processes that a validated 
exemplary project should follow when its staff is preparing for statewi de 
or national diffusion." "We believe that packaging is only one part of the 
total diffusion process . When we speak about packaging we mean the planning 
preparation, production, and putting together of all parts of an effective 
program so that it will then be transportable to other sites. " 
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A BIBLIOGRAPHY OF RESOURCES ON VALIDATION FROM 
THE SEDL/RX REGIONAL RESOURCE CENTER 



A Bouquet of Reading Activities: Pick an Idea . 

Prepared by the Oklahoma Reading Council and Oklahoma Curriculum Improve- 
ment Commission under. the auspices of the Oklahoma State Department of 
Education. Oklahoma State Department of Education. Oklahoma City: 1975. 

This book is "a compilation of descriptions of activities used by teachers 
in Oklahoma." It is a collection of practices, listing objectives, 
materials, procedures and variations and generally comprising less than 
a page / activity. 



Demonstration Schools in Individualized Instruction, 1978-1980 Visitors 

Guide . _ " " 

Texas Education Agency: Division of Dissemination. Austin, Texas: 1978. 

This annual guide supplies a brief overview of Texas' DSH network and 
then lists program descriptions of the 106 state-validated programs by 
Education Service Center region. An index is also available by subject 
and by level . The program descriptions are approximately one page in 
length and include all contact information. Other materials from Texas' 
system will also be available during the conference. 



Getting It All Together,: The JDRP Process . 

Dr. Ann A. Bennett and Dr. Fred L. Fifer. Technical Assistance Brokerage 
(NDN/USOE): Capla Associates. Rochelle Park, New Jersey: 1979. 

Subtitled "a guide for the utilization of the submission process to the 
USOE/NIE Joint Dissemination Review Panel for review and approval of 
exemplary educational programs to be included in the USOE National 
Diffusion Network," this guide book was originally designed as part of a 
TAB package on JDRP submission which includes a pre-post test and a two- 
hour workshop. The book sets out "to explain basic JDRP assumptions and 
clarify subsequent panel processes" to produce a "well formulated, explicit 
step-by-step procedure for assisting school districts wishing to submit 
locally developed educational programs" to JDRP. 

ft- Buide— to Developing Educational Products. 

John J. Ford III, et. al . Developed for the National Learning Disabilities 
Assistance Project under funds provided by Title VI-G of P.L. 91-230. 
NETWORK. Andover, Mass.: 1978. 

"Product development is the process of transforming an initial idea into 
a Usable product. A product is distinguished from other materials by the 
fact that it can be Used without direction or assistance from the. 
developer. This guide is intended to assist in the development of sUch 
products. 11 
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A Hahdbddk for Validating Michigan -Ed ucati onal Prac tices . 

Michigan Department of Education. Lansing, Michigan: March, 1977; 

"This handbook serves to structure the Michigan Validation process for 
all service areas in the Department of Education, n "There are five sec- 
tions to the validation handbook. The introduction contains the purpose 
and definitions, the second section describes the procedure and respon- 
sibilities, the third section is the validation report * the fourth section 
is the onsite visitation form* and the fifth section is the validation 
worksheet. " 



The Joint Dissemination Review Panel IDEABQQK. 

G. Kasten Tallmadge. U.S. HEW: NIE and 0E. Washington: October, 1977. 

The Ideabook is specific to the operation of the Joint Dissemination Review 
Panel and has two purposes : 

1, "To illustrate the many ways to gather convincing evidence 
of the effectiveness of educational innovations . " 

2. "To suggest how to bring current evidence of effectiveness 
together in a succinct and forceful manner." 

Incl uded are many examples and models pertinent to evaluation techniques. 



• packaging— Your Educational Program, Draft Version. 

Fred S. Rosenau and Diane H, Mclntyre. OE: Far West Laboratory. San 
Francisco: January, 1977, 

"This handbook for educators focuses on the processes that a validation 
exemplary project should follow when its staff is preparing for statewide 
or national diffusion." "We believe that packaging is only one part of the 
total diffusion process. When we speak about packaging we mean the planning 
preparation, production, and putting together of all parts of „an effective 
program so that it wil 1 then be transportable to other sites." 

• Pathways to Success : A Resource Manual for the Dissemination of Successful 

Educational Programs . 

Dorothy Soper. New Jersey State Department of Education. Trenton, New 
Jersey: September 1974. 

the manual describes the New Jersey state plan for dissemination of_ 
educational programs. It includes the theoretical rationale as.v/ell^ as 
descriptions of the validation procedure and dissemination requirements. 
Validation in New Jersey requires a commitment by developers to dissemi- 
nation, and dissemination efforts of val idated programs are supported by 
Title IV-C. In addition, programs must be certified as cost-effective 
and their maintenance must not increase a consumer district's per-pupil 
expenditure. Also available will be New Jersey's catalogue of demon- 
stration sites. 
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• The Pennsylvania Diffusion Plan , . Val idated Projects 1978 . 

Prepared for the Bureau of Planning and Evaluation* Pennsylvania Department 
of Education, by Research and information Service for Education. 
(R.I.S.E.j. Pennsylvania Department of Education. King of Prussia, Pa. : 
1978. 

"The projects described in this Booklet were developed by local school 
districts,, intermediate units, and nonpublic schools throughout the 
Commonwealth of Pennsylvania; They were submitted to the Pennsylvania 
Diffusion Panel , whose members reviewed them according to a predetermined 
set of criteria. The Panel is made up of educators from the Pennsylvania 
Department of Education, local school districts and higher education; " 
Materials list projects and give a brief description of the 5-stage 
val idation procedure. 



# A Practical-Jiuixie-±Q- Measuring Project Impact on Student Achievement. 
U.S. HEW: OE. Washington: 1975. 

This guidebook is one of a series of monographs developed by OE "to dis- 
cuss methodological issues in the area of educational program evaluation." 
This particular monograph "deals with only one Central aspect of project 
evaluation, measuring cognitive. achievement gains. . .the entire focus is 
on obtaining as clear and unambiguous an answer as possible to the question , 
"How much more did pupils learn by participating in the project than they 
would have learned without it? ,n "The purpose of the guidebook is to 
provide those concerned with project evaluation with the basic tools they 
need to conduct technically sound, interpretable evaluation studies." 



# A Procedural Guide for Validating Achievement Gains in Educat-ixmal .Projects ; 
U.S. HEW: OE. Washington: 1976. 

This guidebook is one of a series of monographs on Evaluation in Education, 
each intending to represent differing viewpoints. The viewpoint of this 
particular guidebook is "that of the interested party reviewing evaluation 
results and selecting exemplary projects based on them. " "Review and 
appraisal of an evaluations procedures are presented in a series of steps. 
The handbook thus leads the reader systematically to a judgment of whether 
or not the evaluation's results are valid. It also offers suggestions 
for_ correcting those results when certain measurement or analysis prin- 
ciples have been violated." 



• Program Assessment Instruments : Criteria for Excellence. 
Alaska Department of Education. Juneau , Alaska: 1978. 

Materials from the Promising Practices in Education file in the Program 
Development and Dissemination Unit of the Alaska Department of Education. 
Separate program assessment instruments have been developed for each 
program area . Available for your use wi 1 1 be: 

1. Promising Practices in Education Management Format (Record 
Keeping , Awareness, Nomination, Self Assessment, Validation, 
Dissemination) 
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Z. Program Assessment Instruments in Secondary Reading Programs, 
Bilingual-Bicultural Education, District Inservice Program 
Development 

3. Assessment and planning Handbooks in Reading and Math 



• Successful Ventures in Contemporary Education in Oklahoma 1976, Volume XI, 
Selected and arranged by Oklahoma Curriculum Improvement Commission Under 
the auspices of the Curriculum Section. Oklahoma State Department of 
Education. Oklahoma City: 1976. 

"Volume XI of Successful Venturss - in Conte mporary Education in Oklahoma 
is dedicated primarily to reading with some miscel laneous articles." 
'Ventures^ "is a review ofsome most interesting and meaningful innovations 
in the classroom and significant innovations in the teaching and learning 
process. " The book is sponsored by the Oklahoma Curriculum Improvement 
Commission which has as one of its principal functions "the dissemination 
of good educational practices through shared experiences, pooled infor- 
mation , and operative attacks on school problems." 



ADDITIONAL RESOURCES 

• Dr. William Hinze will present the revised edition of Sharing Your 
Success. 

• Ms. Christine Dv/yer will present the Executive Summary of the National 
Association of State Advisory Councils Study of the USOE 1 Identification, 
Val idation, Dissemination ( IVD) Process. 1 

• Other materials will be available such as information about SEDL/RX's 
Soutnwest Educational Program File, Arkansas' Guide for Program 
Selection (validation eligibility), an overview of the val idation 
procedure in Washington State^ and sample catalogs of exemplary programs. 
In addition, fiche of some articles cited in Linda Reed's paper will 

be available, and we'll even have a reader with which to view them! 
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RESOURCES DISTRIBUTEE) SY CONFERENCE PRESENTERS* 



• "The. iVD Process—Myths and Facts Relevant to Prospective Pol icy Directions 
For Exemplary Educational Programs Replication.:. NASAC Study of the USOE 
'Identification, Validation, Dissemination (IVD) Process " Executive 
Summary. National Association of State Advisory Councils. Portsmouth, 
New Hampshire: February 1979. (Distributed by Ms. Christine Dwyer. ) 

• "A Procedural Model for Evaluating Educational Programs in the Grand Rapids 
Public Schools;" Curriculum Planning and Evaluation, Grand Rapids Public 
Schools. Grand Rapids, Michigan: n.d. (Distributed by Dr. dohn Osborne.) 

• "The .Search for Quality Control in Dissemination of -Educational Products 
and Practices: A Look at the Literature and Major Issues." Draft, by Linda 
Reed. CEMREL, Inc. St. Louis, Missouri: April 1979. (Written for the 
conference and distributed April 18th.) 

• "Sharing Educational Success: A Handbook for Validation of Educational 
Practices." Draft. United States Office of Education, National Association 
of State Advisory Council Chairmen, State Departments of Education. 
Washington, D.C.: n.d. (Distributed by Dr. William Hinze.j 

• Packet of sheets from Michigan Department of Education (distributed by Dr. 
John Osborne) : 

. project classification procedure: 

. criteria for project classification 

. experimental and demonstration centers program nomination 
criteria for project classification; 

. project classification abstract; 

. project classification worksheet (experimental); 

. project classification worksheet (demonstration); 

. cnsite visitation form; 

. team visit form. 



*Single copies of these documents are available from: 



Regional Exchange Project 
Southwest Educational .Development Laboratory 
.211 _E. Zth Street. 
Austin j Texas 78/01 
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Resource People 



Ms. Christine Dwyer 

RMC Research Corporation 

111 Bow Street 



Portsmouth, New Hampshire 03801 

Mr. Bud Grossner 
Illinois Office of Education 
100 North First Street 
Springfield, Illinois 62777 

Br. Ben M.. Harris 

Department of Educational Administration 
College of Education 
Education Building 310 
University of Texas at Austin 
Austin, Texas 78712 

Dr. William H. Hinze 
Senior Diffusion Specialist 
HEW/Office of Education 
ROB #3, Room 3606 
7th S^'D" Streets, S.W. 
Washington, D.C. 20202 

Ms. Sharon Koenigs 

CEMREL, Inc. 

3120 59th Street 

St. Louis * Missouri 63139 

Dr. John Osborne 

Michigan Department of Educati on 

P.O. Box 30008 

Lansing, Michigan 48909 



Arkansas 



Mr. John Bess 



State Education Bldg. 
Capitol Mall 



Mr. Mauri ce Dunn 
Mr. Emil Mackey 
Mr. Don Wright 



Little Rock, Arkansas 72201 



Louisiana 



Ms. Daisy Black 
Mr. Dan Carr 



Dr. Ron Dearden 
Dr. Jim Owens 



State Department of Education 



Ms. Yvonne Warner 
Mrs. Sue Wilson 



P.O. Box 44064 
Baton Rouge, Louisiana 70804 



Mississippi 



Mr. Clyde Ratten 



State Department of Education 
P.O. Box 771 

Jackson* Mississippi 39205 
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Ms. Susan Brown 

Mrs. Dolores Bietz State Department of Education 

Ms. Judy Kraft Santa Fe* New Mexico 87503 

Mr. Frank Romero 
Mr. Rehe.Salazar 
Mr. Dan Sanchez 



Oklahoma 



Mrs. Wi Ida Cope] and 
State Department of Education 
Room 3-32, Oliver Hodge Bldg. 
2500 North Lincoln Blvd. 
Oklahoma City, Oklahoma 73105 



Br; Kenneth Eisner 
Department of Education 
Central State University 
Edmond, Oklahoma 73034 



Dr. Amelia Gorena 
State Department of Education 
Room 4-76, Oliver Hodge Bldg. 
2500 North Lincoln Blvd. 
Oklahoma City, Oklahoma 73105 



Dr; Mare Marsden 
1923 Elvin Drive 
Rt. 5, Box 197 
Stillwater, Oklahoma 
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OE Region VI 



Mr. B.F. Peters 
USOE/HEW Region VI 
1200 Main Tower Bldg. 
Dallas, Texas 75282 



Texas 



Ms. Mary Boyvay 
Mrs. Virginia Cutter 
Mr. Pai_Martin 
Mr. Bill Scaririell 
Mc. Ann Shaw 



Texas Education Agency 
201 East ]lth Street 
Austin, Texas 78701 



SEDL/RX Sharon Adams 

~ Nancy Baker Jones 

P res ton C . Kronkos ky 
Jack Lumbley 
Karen 01 sen 
Jan Schechter 
Morrie Schulman 



SEDL 

211 E. 7th Street 
Austin, Texas 78701 
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(Pre-test) . .. ._ . 

VALIDATION CONFERENCE QUESTIONNAIRE 



Please rate statements 1-12 in the following manner: l=strbrigly disagree, 2-disagree, 
3-neutral , 4-agree, 5=strongly agree. 

Rating Statement 



1. An SEA validation procedure is a necessary strategy in advancing 
school i mp rovemen t . 

2. A project, in order to be validated by SEA, must offer "hard evaluation 
data" as an indicator of program effectiveness. 

3. It is the responsibility of the SEA to educate LEA project directors 
concerning the potential hazards in project design and evaluation. 



4. Replicability should be an essential criterion in a validation procedure. 

5. It is absolutely essential that on-site /is its be made to projects 
being considered for validation. 



6. It is the responsibility of the SEA to train validation teams for 
design and evaluation. 

7. SEAs should systematically collect information oh IEA or LEA developed 
promising practices as well as "exemplary" programs. 



8. An SEA validation procedure should groom exemplary programs for federal 
validation procedures. 

9. It is the responsibility of the SEA to support LEA developers in their 
efforts to disseminate their SEA-val idated programs. 

10. An SEA validated program should receive federal funds for dissemination. 

11. I favor the establishment of ah SEA validation procedure for my state. 

12. I favor the establishment of a regional validation procedure to be 
Used on a voluntary basis. 



1.07 
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SS* - 

(Post-test) 

VALIDATION 60NFEREN6n; PARTieiPANT QUESTIONNAIRE 

Please rate statements 1-30 in the following manner: 1=strohg1y disagree, 2=disagree; 
3=neutral, 4=agree, 5=strbrigly agree. 

Rating Statement 

1. I was informed of the objectives of the conference in advance. 

2. The objectives of the conference were appropriate. 

3. The objectives of the eonferenee were achieved. 



4. The conference increased my familiarity with issues, options, concerns, 
and models in validation. 



5. The conference increased my familiarity with validation procedures used 
by the other states in the SEDL/RX region. 



6. The presentations by the resource people at the conference were 
informative. 

7. My SEA may use the services of one of the resource persons from 
the eonferenee in the future. 

8. My SEA team engaged in significant work on our validation concerns 
at the conference. 

9. The material resources provided at the conference were and/or will be 
useful . 

10. The resource person at our SEA team work session(s) provided 
valuable assistance. 

11. Setting aside time to work privately with my SEA team was valuable. 

12. I now have a clearer notion of the ways the SEDL/RX can be of assistance 
to my SEA concerning validation. 

13. The conference was well-organized. 

14. The conference was relevant to my needs. 



15. it is important to exchange educational experiences, information, and 

concerns with other SEAs in the region. 

16. I favor my SEA's participation in a regional program file. 

17. I favor the establishment of a regional validation procedure to be 

used on a voluntary basis. 

18. I would like a follow-up validation conference at the SEA level. 

19. Action which should be taken by my SEA as a result of the conference 

is clear. 
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Eating- (cont. ) Statement 

20. It is the responsibility of the SEA to educate LEA project. di rectors 

concerning the potential hazards in project design and evaluation. 

— — — — 21: It is the responsibility of the SEA to train validation teams for 
design and evaluation; 

22. A project, in order to be validated by an SEA, must offer "hard evaluation 

data" as an indicator of program effectiveness. 

23. SEAs should systematically collect information on ItA or LEA developed 

promising practices as well as "exemplary" programs. 

24. Repl icabi 1 ity should be an essential criterion in a validation procedure. 

25. It is absolutely essential that on-site visits be made to projects 

being considered for validation. 

26. it is the responsibility of the SEA to support LEA developers in 

their efforts to disseminate their SEA-val idated programs. 

27. An SEA validation procedure is a necessary strategy in advancing school 

improvement. 

28. An SEA val idation procedure should groom exemplary programs for 

federal val idation procedures . 

29 . An SEA-val idated program should receive federal funds for dissemination. 

30. I favor the establishment of an SEA validation procedure for my state; 

31. Which State Education Agency or other organization are you representing at the , 
conference? 



32. What was the most valuable aspect of the conference? the least valuable? 
Most valuablej 



Least valuable^- 



33. Please share any other reactions you had to the conference (i.e., perceptions 
regardi ng indi vi dual speakers , the structure of the conference, etc. ) . 



34. As a result of this conference, I will 
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VALIDATION CONFERENCE PRESENTER QUESTIONNAIRE 

Please rate statements 1-16 in the following manner: 1=strong1y disagree, 2=disagree, 
3-heutralj 4=agree, 5=strongly agree. 

Rating Statement 

1. I was informed of the objectives of t h e conference in advance, 

2; The objectives of the conference were appropriate. 

3. The objectives of the conference were achieved. 



4. During conference planning, the type of assistance that I was to provide 

was made clear. 

5. The information that I was asked to present at the conference was 

reJevarit to the heeds of the participants. 

6. The conference helped me gain an understanding of the resources and 

experience of SEA people in the SEDL/RX region. 

7. The conference was well-organized. 

8; The participants at the conference learned new and useful information. 

9. The format of the conference (presentations, individual consultation, 

panel discussion) was effective. 

10. The state team that I worked with engaged in significant work on 

validation concerns. 

11: Which state team did you work with? 



12. How did you feel about performing the various. roles at the conference? 
(i.e.* presenter-, panel member, and individual consultant) 



13. Please share any other reactions you had to the conference; 



tm 
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R£GK)NAL EXCHANGE 

VALIDATION CONFERENCE RECOMMENDATIONS: 

1. What advice would you give to others, especially other SEA 's who want to 
develop a statewide validation procedure? 



2. What resources should be consulted? 



3. What issues must be considered? 



4. What is a workable scope for a statewide validation plan? Should there 
be a limited scope at all? 



5. What essential elements do you advise in a validation procedure? 
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6. What other recommendations do you have? 



Us 
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SEDL/RX REGIONAL VALIDATION CONFERENCE 
FOLLOW-UP 



Have you had occasion to use any of the information or material from the 
Regional Validation Conference? 



□ yes 

□ no 

Please explain 



2. Do you have plans to use any of the information or material from the Reaional 
Validation Conference in the future? 

□ yes 

□ no- 
Please explain 



3. Please comment on the conference paper, "The Search for Quality Control in 

Dissemination of Educational Products and Practices: A took at the Literature 
and Major Issues," written by Linda Reed. 



4. If you have been involved in work on your own state validation plan since the 
conference, have you encountered additional issues, problems, resources, or 
questions you would like to share with us? 

□ yes 

□ no 

If yes, please explain — 



5. Please briefly describe the status of validation work in your state. 



U3 
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6. Has this status changed since the validation conference? If so, can you 
attribute that change to the conference in any way? How? 



Thank you for your time. We appreciated having you at SEDL in Austin, and hope 
to work with you again. If you have any questions or concerns that the SEDL 
Regional Exchange might be able to address, please contact your state's SEDL/RX 
Advisory Board member. 

• Dr. Roland Carpenter, Arkansas ........ 501/371-1287 

• Dr. Ron Dearden, Louisiana 504/342-1151 

• Dr. Jimmy Jones, Mississippi 601/354-7329 

• Mrs. Dolores Dietz, New Mexico 505/827-5441 

• Mr. Jack Craddock, Oklahoma 405/521-3331 

• Mrs. Virginia Cutter, Texas . 512/475-5601 

• Dr. Harold Haswell, HEW/USOE Region VI ... . 214/767-3711 
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REGIONAL VALIDATION CONFERENCE 



Evaluation Findings 



Purpose of Study 

Objectives and expected outcomes for the Regional Validation' Conference, formu- 
lated during the planning and needs assessment stages, are presented on pages 19 and 21. 
The evaluation of the workshop* which consists largely of determining if the objec- 
tives and outcomes were attained, will include the following elements: 

• an analysis of participants 1 responses to a post-workshop questionnaire 
(page 115) designed to measure their impressions of the adequacy of the 
objectives and format of the conference as well as the degree that they 
felt objectives were achieved and expected outcomes were realized; 

• an analysis jf presenters 1 responses to a similar post-workshop 
questionnaire (page 117); 

• an analysis of changes in participants' responses to twelve attitude 
statements which relate to various issues that were highlighted by the 
RDIS synthesis paper on validation. These statements were administered 
prior to the workshop (page 113) and on the post-workshop questionnaire; 
Changes on participants' views on i k ey issues in validation are offered 
as evidence of the impact of the workshop. The presentation of the 
statements prior to the workshop served additional ly as advance organ- 
izers for topics to be addressed at the conference; 

• incoming correspondence regarding the conference; 

• on-going monitoring of the efforts of each state concerning validation 

to determine if the workshop had an influence on the resources the states 
use; 

• a synthesis of recommendations offered by workshop participants. 
Analysis of post-workshop responses: participants 

Twenty individuals responded to the post-workshop questionnaire and rated 30 
statements on the following five-point Likert Scale: l=strongly disagree; 2=disagree; 
3=neutral; 4=agree; 5=strongly agree. Means and standard deviations were computed 
and responses to pertinent questions were analyzed by state to assist in future 
planning. The following results were obtained for those statements having no pre- 
test equivalent: 

• Participants agreed that they were informed of the objectives of 

the conference in advance (x=4;6) , that the_objecti ves were appropriate 
(x=4.6) , and the objectives were achieved (x=4. 1 ) . 




# The majority of participants concluded that the expected outcome 
of increased familiarity with issued, options* concerns* and. .. ... 

models in validation was achieved (x=4.35). A smaller mean (3.8) 
was obtained regarding the second expected outcome of increased 
familiarity with validation procedures used by other statesin the 
SEDb/RX region, Apparently the conference was not as successful 
informing participants of validation procedures used by other states 
in the rlgion as had been desired. This statement recorded the 
lowest mean of any on the questionnaire. 

• Participants responded that the presentations by the resource people 
at the conference were informative (x=4.15) and that the material 
resources provided were or will be useful (x=4^65). Fourteen indivi- 
duals felt their SEA may use the services of one of the resource _ 
persons at the conference in the future (x=4.0), eighteen (x=4.3) felt 
their SEA team engaged in significant work on validation and eighteen 
(x=4.5) felt the resource person at their SEA team sessions provided 
valuable assistance. A mean of 4.4 was obtained for the statement 
"setting aside time to work privately with my SEA team was valuable." 
A breakdown of four of these questions by state is presented in the 

fo : : wing table: 





Mean Rating 




ARK 


hA 


NM 


OKLA 


TX 


My SEA may use the services of one of the 
resource persons from the conference in 
the future. 


3; 33 


5.0 


4.25 


3.25 


3.5 


My SEA team engaged in significant work 
on our validation concerns at the 
conference. 


4.0 


4.8 


4.0 


4.25 


4.5 


The resource person at our SEA team work 
session(s) provided valuable assistance. 


4.0 


5.0 


4.25 


4.5 


4.0 


Setting aside time to work privately 
with my SEA team was valuable. 


4.33 


4.8 


4.25 


4.25 


4.5 



Did participants form a clearer notion of the ways the SEDL/RX can be 
of assistance to their SEA concerning validation as expected? A_mean 
of 3.9 was obtained for this statement; all agreed with the statement 
with the exception of three individuals who gave a neutral response 

The conference was perceived as_wel 1-organized (x=4.25) and relevant. to 
the needs of the participants (x=4.3) . Most respondents favored their 
SEA's participation in a regional program file (x=4. 15) . 



124^10 



ERLC 



# Most voiced support for a follow-up conference at the SEA level 
(x=4;05) arid are clear about action which should be taken by their 
SEA as a result of the conference (5t=4.0). Responses by state to 
the later statements which have implications for future SEDL/RX's 
efforts are included below: . 





Mean Rating 


ARK 


bA 


NM 


OKbA 


TX 


I favor my SEA's participation in a 
regional program file. 


4.0 


4.2 


4.5 


4.5 


4.5 


I would like a follow-up validation 
conference at the SEA-level . 


3.33 


4.8 


3.5 


4.75 


3.0 


Action which should be taken by my SEA 
as a result of the conference is clear. 


3.67 


4.4 


3.0 


4.0 


4.5 



Responses to the post-workshop questionnaire thus indicate the conference was 
successful in meeting specified objectives. The most valuable features of the 
workshop were said to be the presentations of the resource people (Bud Grossner 
was cited by several participants in particular) and the individual state time. 
Most individuals did not identify a least valuable aspect of the conference. 
Virtually all participants responded that as a result of the conference, they will 
work with their states to establish or improve a validation process or framework. 
Analysis of post-workshop responses: presenters 

Three of the four presenters responded to the post-workshop questionnaire 
which contained three open-ended questions and ten statements for each presenter 
to rate on the following five point scale: l=strongly disagree; 2=disagree; 3= 
neutral; 4=agree; 5=strongly agree. 

The presenters responded that: 

• they were informed of the objectives of the conference in advance (x=5) ; 

• the objectives were appropriate (x-4.6); 

• the objectives were achieved (x=4.2); 

• the type of assistance that they were asked to provide was made clear 
during conference planning (x-4.6); 
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• The info inflation they were asked to present was relevant to the 
needs of the pa rtlci pints (x=4.3); 

• The conference helped them gain an Understanding of the resources 
and experience of SEA people in the SEDL/RX region (x=4.3); 

• the conference was well-organized (x=4.6); 

• the participants learned new and useful information (x=4.0); 

• the format (presentations* individual consultation, panel discussion) 
was effective (x=4.3); 

e the state team that they worked with engaged in significant work on 
validation concerns (x=3.8). 

The presenters generally felt comfortable with performing their various roles 
at the conference (presenter, panel member, and individual consultant). One 
presenter reported liking the role of individual consultant the best, while another 
would have preferred more opportunity to offer technical support. 

Other reactions to the conference included the following suggestions: 

• cross team sharing in a small group context; 

• identification* as a large group, of key issues at the beginning of the 
conference; 

• workgroups oriented around a few topics related to validation such as 
training for validators, verification forms, etc.; 

• similar conferences should be performed by other RX's. 
Analysis of pre-post changes^tn- atti tud^^oward^\^idatioa 

At the beginning and at the end of the conference* participants were asked to 
rate twelve attitude statements concerning validation. The five point scale 
described previously was Used (from strongly disagree to strongly agree). 

Pre- and post-workshop questionnaires were matched for fifteen participants. 
A t-test for dependent samples was performed to determine if the conference influenced 
the attitudes of those in attendance; Presumably* if the conference did increase 
familiarity with issues, options* concerns, and models in validation as was expected, 
there should be some changes in participants' attitudes towards those issues * options, 
concerns, and models. A probability of .05 or less was required for significance. 
Table 1 presents the results of this analysis. 
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Table 1 





I 

PRE x 


POST x 


ANALYSIS SUMMARY 


AH SEA validation procedure is a necessary 
strategy in advancing school improvement. 


4.5714 


4.5714 


Pre=post 


A project, in order to be validated by SEA, 
must offer "hard evaluation data" as an 
indicator of program effectiveness. 


3.6429 


4.1429 


Post>Pre* 


It is the responsibility of the SEA to 
educate LEA project directors concerning 
the potential hazards in project design 
and evaluation. 


4.2857 


4.4286 


Post>Pre 


Replicability .hould be an essential 
criterion in a validation procedure. 


4.2143 


4.3571 


Post>Pre 


It is absolutely essential that on-site 
visits be made to projects being 
considered for validation. 


4.4286 


4.5 


Post>Pre= 


It is the responsibility of the SEA to 
train validation teams for design and 
evaluation. 


3.8571 


4.8571 


Post>Pre** 


SEA's should systematically collect 
information on IEA or LEA developed 
promising practices as well as "examplary" 
programs. 


4.0 


4.07 


Post^Pre= 


An SEA validation procedure should groom 
exemplary programs for federal validation 
procedures. 


3.7143 


4.1429 


Post>Pre 


It is the responsibility of the SEA to 
support LEA developers in their efforts to 
disseminate their SEA validated programs. 


4.2143 


4.2857 


Post>Pre= 


An SEA validated program should receive 
federal fund<; for di^^pminriti nri 


■J . 3 / I *T 


o • OD / 1 


Pnc f> P r» 


I favor the establishment of an SEA 
validation for my state. 


4.8571 


4.7857 


Pre>Post= 


I favor. the establishment of a regional 
validation procedure to be used on a 
voluntary basis. 


4.3333 


4.0 


Pre>Post 



* P<-05 Rating scale: l=strongly disagree 

** P<^01 2=dis?nree 
= approximately equal 3=neutral 

4=agree 

5=strohgly agree 
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As can be seen from the chart, after the conference, participants agreed that 
"a project, in order to be validated, must offer 'hard evaluation data' as an 
indicator of program effectiveness" to a significantly greater degree than before 
the conference. Attendants also reported agreeing that "it is the responsibility 
of the SEA to train validation teams for design and evaluation 1 ' significantly more 
following the conference. 

Changes in responses to a third statement regarding the role of SEA validation 
procedures in grooming programs for federal procedures approached significance. 
These changes seem to indicate that the Program Validation Workshop nad an impact 
on the participants. 

incoming correspondence regarding the conference 

(See attarhed copies of letters from: Br. John Osborne, Mr. Dan Carr, Ms. 
Christine Dwyer, and Mr. Dane Man is.) 
Validation Con ference Recommendations: 

In an effort to identify the essential elements of a validation procedure from 
the pool of collective knowledge in the SEDL/RX region regarding validation 
(objective 4), participants at the conference were asked to complete a questionnair 
entitled Validation Conference Recommendations The following is a synthesis of 
responses to the questionnaire. (N=5). The figures in parenthesis refer to the 
number of individuals giving that response. 

1. What advice would you give to others, especially other SEA's who 
want to develop a statewide validation procedure? 

. study present validation procedures in existence (3) 

. identify goals (determine purpose of validation) (3) 

. start small to insure maximum success (1) 

. stick to a strict application of procedures in place (1) 

don't be concerned about assessing potential exportabi 1 i ty (1) 

- get SEA "cabinet" level support for statewide validation 
procedures (1) 
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. think of the. process as dynamic (changeable) hot as 
''the answer." (1) 

2. What resources should be consulted? 

. resource people from other SEA's (4) 
. USOE (2) 

. publications and reports (2) 

. state processes in the region (2) 

. regional labs (1) 

. libraries of colleges and universities (1) 

. consultant firms (1) 

. national validation procedures (1) 

. new and old IVD handbooks (1) 

. the Ideabook (1 ) 

3. What issues must be considered? 

. audience and intended use of the validation procedure (2) 

. the purpose of the validation procedure (3) 

. motivation for validation (1) 

. types of programs to be included (lj 

. the degree of specificity of self-reporting (1) 

; the degree of flexibility in following a validation procedure (lj 

. the scope of validation efforts (1) 

. criteria for validation (1) 

. content of validation (1) 

. dissemination of validated programs (1) 

. motivation for replication (1) 

. extent of replication (1) 

. replication audience: regional or state (1) 
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. individuals to be involved in the validation process (1) 

. the heeds and resources of the state (1) 

What is a workable scope for a statewide validation plan? Should there be 
a limited scope at all? 

. should be unlimited (2) 

• limit to two or three program areas in the first year (1) 

. limit to title programs on a voluntary basis until the 
plan is operational ized and accepted by practitioners (1) 

. include two sets of requirements (national and state) (1) 

What essential elements do you advise in a validation procedure? 

. a carefully stated procedure (2) 

. simple and concise forms (2) 

. training for validators (2) 

. needs assessment to reflect the need for program development (1) 
; planning outline with developmental stages (1) 
. visitation (1) 

. procedure to describe programs (1) 
. detailed effectiveness section (1) 

. criteria that reflect a) minimum standards for the entire 
program, b) values of important components of the program (1) 

. dissemination requirements (1) 

. staff development for adopters/adapters (1) 

. validation lifespan (1) 

. revalidation procedures and processes (1) 

. similar elements to current IVB process (1) 

. strong support at state level (1) 

. workable dissemination network (1) 
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t. What other recommendations do you have? 



. "A follow-up conference with. more concrete direction as to 
what the region plans to do towards meeting objectives of 
validation." (1) 

. "Procedures need to be built in to validate the validation 
plan" (1) 

. "Consultants* linkers ^dissemination specialists, QE, NIE, etc., 
need to address * redefine, and clarify the purpose of validation-- 1 
'to have impact on the education of chi ldren 1 —there has been too 
much emphasis on selling a product. We need to question whether 
this is a necessary or admirable activity." 



ERLC 
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STATE OF MICHIGAN 



"ST- 



JOHN W. PORTER 
Superintendent of 
Public lostrucUon 



DEPARTMENT OF EDUCATION 



tensing, Michigan 48909 



May 21; 19 79 



STATE BOARD OF EDUCATION 

BARBARA .ROBERTS MASON 
President 



NORMAN OTTO STOCKMEYER. SR. 
Vice President 

DR. GUMECINDO SALAS 
Secretary 

B A R B A R A-bUMdUCHELLE 
Treasurer 
JOHN WATANEN, JR. 

NASBE Delegate 

SItVERENIA Q. KANOYTON 

ANNETTA MILLER 
DR. EDMUND F. VANDETTE 



Governor . 

W'LLIAM G, MILLIKEN 
Ex-Officio 



Ms. Jan Schlechter 

D isseminat ion Specialist 

Southwest Educational Development Laboratory 
211 East _7th Street 
Austin Texas 78701 

Dear Jan : 

Just a short note to let you know how pleased I am that 
the participants in your Validation Conference found Michigan's 
procedures to be of interest: My knowledge base was also 
increased as a result of interacting with the presentations and 
the representatives in attendance from several states. 

I enjoyed getting better acquainted with you and other 
staff from the SEDL /Regional Exchange as well as to your mission. 
The Validation Conference was an excellent example of how the 
idea of the exchange concept can be applied. Both you and 
Sharon Koenigs from CEMREL/Regional Exchange, are to be commended 
for your efforts. 




Northwestern State University of Louisiana 

NATCHITOCHES. LOUISIANA 71457 



May 15, 1979 



Ms. dan Schechter 
SEDL/Regional Exchange Project 
211 East 7th Street. 
Austin, Texas 78701 

Dear Jan: 



I just wanted to take a few minutes to express my sincere 
appreciation for all you and the others at SEDL did for the 
group at the recen t yal i da t i on con f e rence .1 enjoyed every 
moment of the conference and profited immensely from it . You 
had the best conference that I have ever had the privilege of 
attending; My compliments to you and everyone else involved. 

I have read Linda Reed's paper and found it well organized and 
very informative* As a person "new" to the national and state 
validation process and inherent problems, I found the transitions 
from topic to topic somewhat confusing and difficult to follow, 
mainly due to my inadequate background^ I am not sure what 
audience the paper is prepared for, but anticipating a broad 
population of interest, the definition of some terms used would 
be advantageous to the reader; Otherwise, I feel Linda has 
done an exceptional job. I do hope that when the paper is 
finalized, I will have the opportunity to review that manuscript. 



Again, please accept my thanks for a job well done. Extend to 
all the members of SEDL my deep appreciation for all they did 
to make the conference the .success that it was. I hope that 




Dan B. Carr 

DBC:es 
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RMC Eesearefj Corporation 



1 1 1 Bow Street Portsmouth. New Hampshire 03801 * Telephone: (603) 436-5385 436-5381 



Jan Schechter 
Southwest Educational 

Deve lopment Laboratory 
211 East 7th Street 
Austin, Texas 78701 

Dear Jan: 

Just a brief note to let you know that 
I en j oyed my participation in the recent 
validation conference . I especially 
appreciate all the things you did to make 
the two days easy and pleasant for me. 

I also enclose my brief responses to 
the questionnaire on recommendations • 

I hope we wil 1 have the opportunity to 
work together again. 



April 27, 1979 




Christine Dwyer 



Research Associate 



CD/ Raw 
Enclosure 
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An Educational Laboratory 



Preston Kronkosky, Director 
SEDL Regional Exchange 
211 E; Seventh Street 
Austin, Texas 78701 



April 30, 1979 




Dear Preston: 

It was certainly our pleasure to participate with you in the SEDL 
Validation Conference April 19 and 20. Sharon reported that it was an 
excellent, well-planned meeting. It serves as a fine example of 
collaboration across RDx contracts, educational laboratories, and state 
departments of education. We look forward to additional opportunities 
for collaborative work in the future. 

Enclosed are copies of letters sent to Bud Grossner and John Osborne for 
your records. As agreed, we are supporting their participation in the 
conference. 

Again, thank you for the opportunity; We look forward to seeing you at 
the RDx meeting in a week. 



Sincerely, 




Dane Man is, 
Director 



Midwest Regional Exchange 




Sharon S. Koenigs 
Resource Coordi nator 
Midwest Regional Exchange 



air 
Enc. 




3120 59th Street 
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SEDL/RX REGIONAL VALIDATION CONFERENCE 
Follow-up Study 



Twenty-six individuals attended the SEDL/RX Validation Conference. Each 

delegation responded to a follow-up questionnaire distributed to participants 14 

weeks after the conference and designed to examine the impact of the two day meeting. 

A total of twelve questionnaires representing six states and the Office of Education 

were returned, two of which appear to represent the responses of entire state groups; 

Reactions to the six questions on the follow-up instrument are presented below. 

2. Have you had occasion to use any of the information or material from the Regional 
Validation Conference? 

At least one participant from each state responded affirmatively to this 
question; a total of ten positive responses were obtained. Participants 
used the information or materials from the conference in the following ways: 

. in developing a draft validation plan; 

. in revising state val idation plan ; 



. in assisting school districts in planning programs for future val idation; 

. by distributing copies of materials to the director of federal programs, 
Title IV-6 and to the state facilitator; 

. as resources for a committee review of the state validation process; 

. in gaining perspective on what other states are doing in the area of 
val idation (i.e., criteria and procedures to val i date programs) and/or 
OE aims and policies; 

. by sharing the information with other SDE personnel ; 

. by sharing the information with others in the Office of Education. 

Two participants from different states reported that they have not been invol ved 
in nor to their knowledge have there been any meetings to establish state 
validation plans. 

2 . Do you have plans to use any of the information or material from the Regional 
Valida tion Conference in the future? 

Al 1 twelve questionnaires contained positive responses. Participants intend to 
use the information and materials in planning state validation processes and 
model s^ in comparing their state plans with other states, in adjusting forms and 
on-site visits, in obtaining resource people in the future, and in providing a 
bibliography of resources to others in the State Department of Education. 
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Please comment on the conference paper "The Search for Quality Control in 
Dissemination of Educational Products and Practices: A Look at the Literature 
and Major Issues , " written by Linda Reed. 

All returning questionnai res praised Ms. Reed's paper. Characteristic comments 
were: 

. "...A very comprehensive review of procedures which have been/are 
being utilized. The concerns addressed are very relevant," 

. ''Helpful in understanding ways of gathering evidence of success 
and acceptable kinds of validation results " 

. "The paper is well written; We will be using many of the approaches 
described as we start to look at validation...." 

. "Excellent paper that will be useful to us. ... " 

. "(it) Gave a good overview of the state of validation and some of the 
problems that need to be resolved." 

If you have been involved in work on your own state validation plan since the 
conference , have you encountered addi tional issues , problems f resources , or 
questions you would like to share with us? 

Six positive responses were obtained for this question ; participants cited the 
following areas of concern: 

. the degree of involvement that the SDE will have and when it will occur; 

, the importance of monitoring programs before they are submitted for 
validation to ensure that proper statistics will be acquired; 

. the difficulty in evaluating non-classroom types of programs and 
programs covering multiple academic areas of which include both staff 
development and classroom instruction. 

Please briefly describe the status of validation work in your state. 

The six states in the SEDL/RX region are in varying stages of involvement in 
validation; One state reports being in the early stages of work in validation. 
Another has formed a subcommittee which has been receiving information and 
materials on validation and will be staffed full-time next year to devise a 
state plan. A third state reports a draft model and process for validation 

are ready for review. Another uses the current IVD process while a_f if th 

uses IVD materials and validators trained by OE. The remaining state in the 
region is substantially revising its process to expand into all areas of 
classroom instructional programs and non-classroom programs such as staff 
development, school communication, guidance and counseling, learning resource 
centers , and community education. 
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6. Has this status changed since the validation conference? if so, can you 
attribute that change to the conference is any way? How? 

Three states report no changes since the conference. A fourth reports no 
substantivechanges at the current time, although materials from the conference 
generated discussion between departments and a written plan now exists where 
hone. did before. Another state claims that as a result of the conference, 
considerable progress has been made in developing procedures for state valida- 
tion. The remaining state in the region reports a change in status, "but the 
impetus and efforts had already begun." 



Cbhcl us ion 

The responses of participants who attended the Regional Validation Conference 
to a follow-up questionnaire reveal that useful information and material were imparted 
at the conference. At the least, the workshop served to stimulate discussion of 
validation within the State Departments of Education in the SEDL/RX region and within 
the Office of Education. Eaeh state has found the information from the conference 
useful and all who completed the questionnaire plan to use the information or 
materials in the future. Two states report a change of status in validation work 
since the conference, and one attributes progress in developing a state validation 
plan to the validation conference. 
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REPORT BY DR. BEN HARRIS: POST CONFERENCE OBSERVATIONS AND RECOMMENDATIONS 

Dr: Ben Harris, professor of educational, administration at The University 
of Texas at Austin, served the SEDL/RX as an objective participant/observer. It 
was his responsibility to attend the conference, sit in on each state's team 
session as an unobtrusive observer, and, in general, provide a perspective about 
how the SEA's were similar, different, what any underlying themes were, and 
recommend how the SEDL/RX might further aid the SEA's in their work toward state- 
wide validation procedures. On May 11 , Dr. Harris met with the SEDL/RX staff and 
offered the following observations and recommendations. 
Observations 

In general, Harris said the conference was a good set of experiences, that 
participants were appreciative of the opportunity to share with each other and to 
get input from others. Its problem, as usual, was time. The conference was too 
short and he observed some participants feeling rushed and attempting to go back 
home with something. The danger with this> Harris said* was that the product 
might not hold up. He also recommended more interaction among participants earlier* 
to allow them time to sit in mixed groups and trade ''war stories" and rationale. 

Dr. Harris also made specific observations about each state. Arkansas is 
involving more state-level and out-of-state people in its validation process than 
LEA or "grass roots" people, though the state's process is itself well structured, 
Harris commented that involvement was limited to a few and that only national-calibre 
programs were considered, to the exclusion of state-level or district-level programs. 

Oklahoma and New Mexico each were moving toward the restrictions of a 
paper rather than program file, using descriptions from the district rather than 
bri-site visits. Both states needed to consider the impact of such paper- 
c entered files on the validation process in contrast with people-centered valida- 
tions. How can such programs be fairly or completely pictured on paper? 
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And what if poor programs have excellent writers, or excellent programs poor 
writers? The document collection route puts a great deal of faith in the 
document* and this may be unwise, especially if one later wishes to parlay 
the collection into something more valuable, more descriptive. Oklahoma, he felt, 
needed a clear set of effectiveness criteria and one unified set of state procedures 
instead of the current tendency for each LEA department to use its own. (NOTE: 
In late July, the SEDL/RX staff learned that New Mexico had changed its perspective 
and added an on-site visit component to the draft of its validation manual.) 

Louisiana appeared interested in the Illinois model , an extensive process which 
makes use of regional education service centers throughout the state. A state like 
Louisiana, which does not already have a regional service center system, would need 
to consider large scale reorganization to adopt a model which depends on them. 

Texas ' established plan was advanced to the point that the team was ready 
to work on a rationale for expanding it to include new program areas such as 
inservice education as well as instructional techniques. Because of the state's 
long experience with a validation procedure , Harris felt it needed the least 
guidance and observation, although it could use a bit more paper file documentation. 

Mississippi sent only one SEA observer to the conference as this was its second 
activity with the SEDL/RX (the first had been an Advisory Board meeting the day before) 
and because the SEA had not joined the SEDL/RX service area unti 1 conference planning 
was nearly finished. 
Recommendations for SEDL/RX 

• Consider developing training programs for validation, perhaps creating 
training modules which would allow participants to analyze and sort out 
good, poor, and unclear data. Other topics could be "how to use ah . 
observation instrument," "how to construct an evaluation instrument* 11 
"what to look for on a site visi t, 11 and "how to handle slippery data. " 

• Consider how the SEDL/RX staff can help the states to think carefully about 
what they are designing, to ask them first to define what they want in their 
final product. 
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• The validation process begins w|th the completion of a developmental 
design, then moves into ah experimental or field-test stage. A program 
can then be attempted by bthor sites^ .during which various elements of 
the design eari be monitored for adoption and adaptation. After this 
information is collected, project designers may. then be able to sift the 
elements of the program which are essential to its success. In light of 
this process, consider the basic elements of any program— its skeleton; 
What. is this thing that gives the program its UhigUe structure? What 
are the core elements? What purpose does the skeleton serve? Not 
serve? What supports it? And what contraindications . are there? 

Unde^ what conditions will it and will it not work? Might it be the 
case that we can only specify when a program does not Work, or is not 
in place? 

• Follow up this conference with state-wide conferences in which regions 

of the state could come together to _discuss_the_state process, experience 
a mock validation session to learn first hand what is necessary for 
sueeess, and consider what they would want to see at an on-site visit, 

• Consider preparing a synthesis of effective inservice techniques for use 
in implementing new programs. Such a synthesis would help SEA's aid 
LEA's to design inservice to fit their needs as nearly as possible, and 
would also be a valuable resource for any educator interested in 
implementing change. 

A General Cautio n 

What teachers do with students and materials and equipment to facilitate 
learning is the heart of any program of instruction. Many programs tend to be 
defined and described and hence validated in terms of what is taught, how students 
respond, or what is learned. Important as these may be as ways of detailing 
program realities, diffusion depends on describing a way of operating and especially 



on ways teachers perform. No validations that neglect this are likely to be very 
useful . 




